Uses of Class
net.sf.okapi.steps.tokenization.common.Lexems

Packages that use Lexems

Package	Description
net.sf.okapi.steps.tokenization.common
net.sf.okapi.steps.tokenization.engine

Uses of Lexems in net.sf.okapi.steps.tokenization.common

Methods in net.sf.okapi.steps.tokenization.common that return Lexems

Modifier and Type	Method and Description
Lexems	ILexer.process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.

Uses of Lexems in net.sf.okapi.steps.tokenization.engine

Methods in net.sf.okapi.steps.tokenization.engine that return Lexems

Modifier and Type	Method and Description
Lexems	TokenScanner.process(String text, LocaleId language, Tokens tokens)
Lexems	TextScanner.process(String text, LocaleId language, Tokens tokens)
Lexems	GarbageCollector.process(String text, LocaleId language, Tokens tokens)
Lexems	LookUpLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	Reconciler.process(String text, LocaleId language, Tokens tokens)
Lexems	TokenMatcher.process(String text, LocaleId language, Tokens tokens)
Lexems	JavaCcLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	RbbiLexer.process(String text, LocaleId language, Tokens tokens)
Lexems	Sorter.process(String text, LocaleId language, Tokens tokens)
Lexems	Remover.process(String text, LocaleId language, Tokens tokens)
Lexems	Retainer.process(String text, LocaleId language, Tokens tokens)