Uses of Class
net.sf.okapi.steps.tokenization.tokens.Tokens

Packages that use Tokens

Package	Description
net.sf.okapi.steps.tokenization
net.sf.okapi.steps.tokenization.common
net.sf.okapi.steps.tokenization.engine
net.sf.okapi.steps.tokenization.tokens

Uses of Tokens in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Tokens

Modifier and Type	Method and Description
static Tokens	Tokenizer.tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	Tokenizer.tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	Tokenizer.tokenizeString(String text, LocaleId language, String... tokenNames) Extracts tokens from the given text.

Uses of Tokens in net.sf.okapi.steps.tokenization.common

Methods in net.sf.okapi.steps.tokenization.common that return Tokens

Modifier and Type	Method and Description
Tokens	TokensAnnotation.getFilteredList(String... tokenTypes)
Tokens	TokensAnnotation.getTokens()

Methods in net.sf.okapi.steps.tokenization.common with parameters of type Tokens

Modifier and Type	Method and Description
void	TokensAnnotation.addTokens(Tokens tokens)
protected abstract void	AbstractLexer.lexer_open(String text, LocaleId language, Tokens tokens)
void	AbstractLexer.open(String text, LocaleId language, Tokens tokens)
void	ILexer.open(String text, LocaleId language, Tokens tokens) Starts processing a string or a list of tokens, extracting lexems from them.
Lexems	ILexer.process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.
void	TokensAnnotation.setTokens(Tokens tokens)

Constructors in net.sf.okapi.steps.tokenization.common with parameters of type Tokens

Constructor and Description
TokensAnnotation(Tokens tokens)

Uses of Tokens in net.sf.okapi.steps.tokenization.engine

Methods in net.sf.okapi.steps.tokenization.engine with parameters of type Tokens

Modifier and Type	Method and Description
protected void	Reconciler.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	TextScanner.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	Sorter.lexer_open(String text, LocaleId language, Tokens tokens)
void	Retainer.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	JavaCcLexer.lexer_open(String text, LocaleId language, Tokens tokens)
void	Remover.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	TokenScanner.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	LookUpLexer.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	GarbageCollector.lexer_open(String text, LocaleId language, Tokens tokens)
protected void	TokenMatcher.lexer_open(String text, LocaleId language, Tokens tokens)
void	RbbiLexer.lexer_open(String text, LocaleId language, Tokens tokens)
Lexems	Reconciler.process(String text, LocaleId language, Tokens tokens)
Lexems	TextScanner.process(String text, LocaleId language, Tokens tokens)
Lexems	Sorter.process(String text, LocaleId language, Tokens tokens)
Lexems	Retainer.process(String text, LocaleId language, Tokens tokens)
Lexems	JavaCcLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	Remover.process(String text, LocaleId language, Tokens tokens)
Lexems	TokenScanner.process(String text, LocaleId language, Tokens tokens)
Lexems	LookUpLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	GarbageCollector.process(String text, LocaleId language, Tokens tokens)
Lexems	TokenMatcher.process(String text, LocaleId language, Tokens tokens)
Lexems	RbbiLexer.process(String text, LocaleId language, Tokens tokens)

Uses of Tokens in net.sf.okapi.steps.tokenization.tokens

Methods in net.sf.okapi.steps.tokenization.tokens that return Tokens

Modifier and Type	Method and Description
Tokens	Tokens.getFilteredList(String... tokenNames) Return a list of Token objects.