Package net.sf.okapi.steps.tokenization

Class Tokenizer

java.lang.Object

net.sf.okapi.steps.tokenization.Tokenizer

public class Tokenizer
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
protected static TokenizationStep	ts

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
static Tokens	tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	tokenizeString(String text, LocaleId language, String... tokenNames)	Extracts tokens from the given text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ts

protected static TokenizationStep ts

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

tokenizeString

protected static Tokens tokenizeString(String text,
                                       LocaleId language,
                                       String... tokenNames)

Extracts tokens from the given text.

Parameters:

text - Text to tokenize.

language - Language of the text.

tokenNames - Optional list of token names. If omitted, all tokens will be extracted.

Returns:

A list of TokenType objects.

tokenize

public static Tokens tokenize(ITextUnit textUnit,
                              LocaleId language,
                              String... tokenNames)

tokenize

public static Tokens tokenize(TextContainer textContainer,
                              LocaleId language,
                              String... tokenNames)

tokenize

public static Tokens tokenize(TextFragment textFragment,
                              LocaleId language,
                              String... tokenNames)

tokenize

public static Tokens tokenize(String string,
                              LocaleId language,
                              String... tokenNames)