Package opennlp.tools.tokenize
Interface Detokenizer
-
- All Known Implementing Classes:
DictionaryDetokenizer
public interface DetokenizerADetokenizermerges tokens back to their detokenized representation.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static classDetokenizer.DetokenizationOperationThis enum contains an operation for every token to merge the tokens together to their detokenized form.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description Detokenizer.DetokenizationOperation[]detokenize(String[] tokens)Detokenizes the collection of tokens.Stringdetokenize(String[] tokens, String splitMarker)Detokenizes the inputtokensinto a String.
-
-
-
Method Detail
-
detokenize
Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
Detokenizes the collection of tokens.- Parameters:
tokens- The elements which should be detokenized.- Returns:
- The
merge operationsto handle giventokens.
-
detokenize
String detokenize(String[] tokens, String splitMarker)
Detokenizes the inputtokensinto a String. Tokens which are connected without awhitespacecharacter in between can be separated by a givensplitMarker.- Parameters:
tokens- The elements which should be concatenated.splitMarker- The split marker ornull.- Returns:
- The concatenated tokens as a single string.
-
-