smile.nlp.tokenizer

Interface Tokenizer

All Known Implementing Classes:

BreakIteratorTokenizer, PennTreebankTokenizer, SimpleTokenizer

public interface Tokenizer

A token is a string of characters, categorized according to the rules as a symbol. The process of forming tokens from an input stream of characters is called tokenization.

This is not as easy as it sounds. For example, when should a token containing a hypen be split into two or more tokens? When does a period indicate the end of an abbreviation as opposed to a sentence or a number or a Roman numeral? Sometimes a period can act as a sentence terminator and an abbreviation terminator at the same time. When should a single quote be split from a word?

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	split(java.lang.String text) Splits the string into a list of tokens.

Method Detail

split

java.lang.String[] split(java.lang.String text)

Splits the string into a list of tokens.