Uses of Class
net.sf.okapi.steps.tokenization.Tokens

Uses of Tokens in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Tokens

Modifier and Type	Method	Description
Tokens	Tokens.getExcludedFilteredList(String... tokenNames)	Return a list of TokenType objects.
Tokens	Tokens.getFilteredList(String... tokenNames)	Return a list of TokenType objects.
Tokens	TokensAnnotation.getFilteredList(String... tokenTypes)
Tokens	TokensAnnotation.getTokens()
static Tokens	Tokenizer.tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	Tokenizer.tokenizeString(String text, LocaleId language, String... tokenNames)	Extracts tokens from the given text.

Methods in net.sf.okapi.steps.tokenization with parameters of type Tokens

Modifier and Type	Method	Description
void	TokensAnnotation.addTokens(Tokens tokens)
void	TokensAnnotation.setTokens(Tokens tokens)

Constructors in net.sf.okapi.steps.tokenization with parameters of type Tokens

Constructor	Description
TokensAnnotation(Tokens tokens)