net.sf.okapi.steps.tokenization.engine.javacc

Class WordTokenizer

java.lang.Object

net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

All Implemented Interfaces:

WordTokenizerConstants

public class WordTokenizer
extends Object
implements WordTokenizerConstants

A grammar-based tokenizer constructed with JavaCC.

This should be a good tokenizer for most European-language documents.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

Field Summary

Fields

Modifier and Type	Field and Description
Token	jj_nt Next token.
Token	token Current token.
WordTokenizerTokenManager	token_source Generated Token Manager.

Fields inherited from interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

ACRONYM, ALPHA, ALPHANUM, APOSTROPHE, COMPANY, DEFAULT, DIGIT, EMAIL, EOF, HAS_DIGIT, HOST, HYPHENATED, LETTER, NOISE, NUM, P, tokenImage

Constructor Summary

Constructors

Constructor and Description
WordTokenizer(InputStream stream) Constructor with InputStream.
WordTokenizer(InputStream stream, String encoding) Constructor with InputStream and supplied encoding
WordTokenizer(Reader stream) Constructor.
WordTokenizer(WordTokenizerTokenManager tm) Constructor with generated Token Manager.

Method Summary

Modifier and Type	Method and Description
void	disable_tracing() Disable tracing.
void	enable_tracing() Enable tracing.
ParseException	generateParseException() Generate ParseException.
Token	getNextToken() Get the next Token.
Token	getToken(int index) Get the specific Token.
Token	nextToken() Returns the next token in the stream, or null at EOS.
void	ReInit(InputStream stream) Reinitialise.
void	ReInit(InputStream stream, String encoding) Reinitialise.
void	ReInit(Reader stream) Reinitialise.
void	ReInit(WordTokenizerTokenManager tm) Reinitialise.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

token_source

public WordTokenizerTokenManager token_source

Generated Token Manager.

token

public Token token

Current token.

jj_nt

public Token jj_nt

Next token.

Constructor Detail

WordTokenizer

public WordTokenizer(InputStream stream)

Constructor with InputStream.

WordTokenizer

public WordTokenizer(InputStream stream,
                     String encoding)

Constructor with InputStream and supplied encoding

WordTokenizer

public WordTokenizer(Reader stream)

Constructor.

WordTokenizer

public WordTokenizer(WordTokenizerTokenManager tm)

Constructor with generated Token Manager.

Method Detail

nextToken

public final Token nextToken()
                      throws ParseException,
                             IOException

Returns the next token in the stream, or null at EOS.

The returned token's type is set to an element of WordTokenizerConstants.tokenImage.

Throws:

ParseException

IOException

ReInit

public void ReInit(InputStream stream)

Reinitialise.

ReInit

public void ReInit(InputStream stream,
                   String encoding)

Reinitialise.

ReInit

public void ReInit(Reader stream)

Reinitialise.

ReInit

public void ReInit(WordTokenizerTokenManager tm)

Reinitialise.

getNextToken

public final Token getNextToken()

Get the next Token.

getToken

public final Token getToken(int index)

Get the specific Token.

generateParseException

public ParseException generateParseException()

Generate ParseException.

enable_tracing

public final void enable_tracing()

Enable tracing.

disable_tracing

public final void disable_tracing()

Disable tracing.