com.aliasi.test.unit.tokenizer

Class ConstantTokenizer

java.lang.Object

com.aliasi.tokenizer.Tokenizer

com.aliasi.test.unit.tokenizer.ConstantTokenizer

All Implemented Interfaces:

Iterable<String>

public class ConstantTokenizer
extends Tokenizer

Constructor Summary

Constructors

Constructor and Description
ConstantTokenizer(String[] tokens, String[] whitespaces)

Method Summary

Methods

Modifier and Type	Method and Description
String	nextToken() Returns the next token in the stream, or null if there are no more tokens.
String	nextWhitespace() Returns the next whitespace.

Methods inherited from class com.aliasi.tokenizer.Tokenizer

iterator, lastTokenEndPosition, lastTokenStartPosition, tokenize, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConstantTokenizer

public ConstantTokenizer(String[] tokens,
                 String[] whitespaces)

Method Detail

nextToken

public String nextToken()

Description copied from class: Tokenizer

Returns the next token in the stream, or null if there are no more tokens. Flushes any whitespace that has not been returned.

Specified by:

nextToken in class Tokenizer

Returns:

The next token, or null if there are no more tokens.

nextWhitespace

public String nextWhitespace()

Description copied from class: Tokenizer

Returns the next whitespace. Returns the same result for subsequent calls without a call to nextToken.

The default implementation in this class is to return a single space, Strings.SINGLE_SPACE_STRING.

Overrides:

nextWhitespace in class Tokenizer

Returns:

The next space.