net.sf.okapi.steps.tokenization

Interface ITokenizer

All Known Implementing Classes:

RbbiTokenizer

public interface ITokenizer

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	ITokenizer.TokenType

Method Summary

Modifier and Type	Method and Description
boolean	hasNext()
void	init(String text, LocaleId language)
Token	next()

Method Detail

hasNext

boolean hasNext()

next

Token next()

init

void init(String text,
          LocaleId language)