banner.tokenization

Class SimpleTokenizer

java.lang.Object

banner.tokenization.SimpleTokenizer

All Implemented Interfaces:

Tokenizer

public class SimpleTokenizer
extends Object
implements Tokenizer

Tokens ouput by this tokenizer consist of a contiguous block of alphanumeric characters or a single punctuation mark. Note, therefore, that any construction which contains a punctuation mark (such as a contraction or a real number) will necessarily span over at least three tokens.

Author:

Bob

Constructor Summary

Constructors

Constructor and Description
SimpleTokenizer()

Method Summary

Modifier and Type	Method and Description
List<String>	getTokens(String text)
void	tokenize(Sentence sentence) Tokenizes the given Sentence

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleTokenizer

public SimpleTokenizer()

Method Detail

tokenize

public void tokenize(Sentence sentence)

Description copied from interface: Tokenizer

Tokenizes the given Sentence

Specified by:

tokenize in interface Tokenizer

Parameters:

sentence - The Sentence to tokenize

getTokens

public List<String> getTokens(String text)

Specified by:

getTokens in interface Tokenizer