Uses of Class
net.sf.okapi.steps.tokenization.Tokens

Uses of Tokens in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Tokens

Modifier and Type	Method and Description
Tokens	Tokens.getExcludedFilteredList(String... tokenNames) Return a list of TokenType objects.
Tokens	Tokens.getFilteredList(String... tokenNames) Return a list of TokenType objects.
Tokens	TokensAnnotation.getFilteredList(String... tokenTypes)
Tokens	TokensAnnotation.getTokens()
static Tokens	Tokenizer.tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	Tokenizer.tokenizeString(String text, LocaleId language, String... tokenNames) Extracts tokens from the given text.

Methods in net.sf.okapi.steps.tokenization with parameters of type Tokens

Modifier and Type	Method and Description
void	TokensAnnotation.addTokens(Tokens tokens)
void	TokensAnnotation.setTokens(Tokens tokens)

Constructors in net.sf.okapi.steps.tokenization with parameters of type Tokens

Constructor and Description
TokensAnnotation(Tokens tokens)