Uses of Class
net.sf.okapi.steps.tokenization.Token

Uses of Token in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Token

Modifier and Type	Method and Description
Token	RbbiTokenizer.next()
Token	ITokenizer.next()

Methods in net.sf.okapi.steps.tokenization that return types with arguments of type Token

Modifier and Type	Method and Description
List<Token>	TokenizationStep.apostrophe(Token token, LocaleId locale) Break French and Italian words with apostrophe into three tokens WORD, PUNCTUATION, WORD
Collection<? extends Token>	TokenizationStep.postProcess(Token t, LocaleId language) Various rules to make corrections to RbbiTokenizer

Methods in net.sf.okapi.steps.tokenization with parameters of type Token

Modifier and Type	Method and Description
List<Token>	TokenizationStep.apostrophe(Token token, LocaleId locale) Break French and Italian words with apostrophe into three tokens WORD, PUNCTUATION, WORD
Collection<? extends Token>	TokenizationStep.postProcess(Token t, LocaleId language) Various rules to make corrections to RbbiTokenizer