|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | ||||||||
public interface Detokenizer
A Detokenizer merges tokens back to their untokenized representation.
| Nested Class Summary | |
|---|---|
static class |
Detokenizer.DetokenizationOperation
This enum contains an operation for every token to merge the tokens together to their detokenized form. |
| Method Summary | |
|---|---|
Detokenizer.DetokenizationOperation[] |
detokenize(String[] tokens)
Detokenize the input tokens. |
String |
detokenize(String[] tokens,
String splitMarker)
Detokenize the input tokens into a String. |
| Method Detail |
|---|
Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
tokens - the tokens to detokenize.
String detokenize(String[] tokens,
String splitMarker)
tokens - splitMarker - the split marker or null
|
|||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | ||||||||