| Package | Description |
|---|---|
| net.sf.okapi.steps.tokenization.engine.javacc |
| Modifier and Type | Method and Description |
|---|---|
ParseException |
WordTokenizer.generateParseException()
Generate ParseException.
|
| Modifier and Type | Method and Description |
|---|---|
Token |
WordTokenizer.nextToken()
Returns the next token in the stream, or null at EOS.
|
Copyright © 2018. All rights reserved.