banner.tokenization

Interface Tokenizer

All Known Implementing Classes:

SimpleTokenizer

public interface Tokenizer

Tokenizers take the text of a sentence and turn it into a Sentence object. They do not fill in the Mentions in the sentence, and the Tokens which make up the sentence have no fields besides the text field filled in.

Method Summary

Modifier and Type	Method and Description
List<String>	getTokens(String text)
void	tokenize(Sentence sentence) Tokenizes the given Sentence

Method Detail

tokenize

void tokenize(Sentence sentence)

Tokenizes the given Sentence

Parameters:

sentence - The Sentence to tokenize

getTokens

List<String> getTokens(String text)