SimpleTokenizer (JCoRe Banner 2.6.1 API)

java.lang.Object
- banner.tokenization.SimpleTokenizer

All Implemented Interfaces:

Tokenizer
```
public class SimpleTokenizer
extends Object
implements Tokenizer
```
Tokens ouput by this tokenizer consist of a contiguous block of alphanumeric characters or a single punctuation mark. Note, therefore, that any construction which contains a punctuation mark (such as a contraction or a real number) will necessarily span over at least three tokens.

Author:

Bob

Constructor Summary

Constructors
Constructor Description

SimpleTokenizer()

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`List<String>`	`getTokens(String text)`
`void`	`tokenize(Sentence sentence)`	Tokenizes the given `Sentence`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - SimpleTokenizer
```
public SimpleTokenizer()
```
- Method Detail
  - tokenize
```
public void tokenize(Sentence sentence)
```
    Description copied from interface: Tokenizer
    
    Tokenizes the given Sentence
    
    Specified by:
    
    tokenize in interface Tokenizer
    
    Parameters:
    
    sentence - The Sentence to tokenize
  - getTokens
```
public List<String> getTokens(String text)
```
    Specified by:
    
    getTokens in interface Tokenizer