Tokenizer (JCoRe Banner 2.6.0 API)

All Known Implementing Classes:

SimpleTokenizer
```
public interface Tokenizer
```
Tokenizers take the text of a sentence and turn it into a Sentence object. They do not fill in the Mentions in the sentence, and the Tokens which make up the sentence have no fields besides the text field filled in.

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method	Description
`List<String>`	`getTokens(String text)`
`void`	`tokenize(Sentence sentence)`	Tokenizes the given `Sentence`

- Method Detail
  - tokenize
```
void tokenize(Sentence sentence)
```
    Tokenizes the given Sentence
    
    Parameters:
    
    sentence - The Sentence to tokenize
  - getTokens
```
List<String> getTokens(String text)
```