Package banner.tokenization
-
Interface Summary Interface Description Tokenizer Tokenizers take the text of a sentence and turn it into a Sentence object. -
Class Summary Class Description SimpleTokenizer Tokens ouput by this tokenizer consist of a contiguous block of alphanumeric characters or a single punctuation mark.