net.sf.okapi.steps.tokenization.engine.javacc

Class WordTokenizerTokenManager

java.lang.Object

net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

All Implemented Interfaces:

WordTokenizerConstants

public class WordTokenizerTokenManager
extends Object
implements WordTokenizerConstants

Token Manager.

Field Summary

Fields

Modifier and Type	Field and Description
protected char	curChar
PrintStream	debugStream Debug output.
protected SimpleCharStream	input_stream
static String[]	jjstrLiteralImages Token literal values.
static String[]	lexStateNames Lexer state names.

Fields inherited from interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

ACRONYM, ALPHA, ALPHANUM, APOSTROPHE, COMPANY, DEFAULT, DIGIT, EMAIL, EOF, HAS_DIGIT, HOST, HYPHENATED, LETTER, NOISE, NUM, P, tokenImage

Constructor Summary

Constructors

Constructor and Description
WordTokenizerTokenManager(SimpleCharStream stream) Constructor.
WordTokenizerTokenManager(SimpleCharStream stream, int lexState) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
Token	getNextToken() Get the next Token.
protected Token	jjFillToken()
void	ReInit(SimpleCharStream stream) Reinitialise parser.
void	ReInit(SimpleCharStream stream, int lexState) Reinitialise parser.
void	setDebugStream(PrintStream ds) Set debug output.
void	SwitchTo(int lexState) Switch to specified lex state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

debugStream

public PrintStream debugStream

Debug output.

jjstrLiteralImages

public static final String[] jjstrLiteralImages

Token literal values.

lexStateNames

public static final String[] lexStateNames

Lexer state names.

input_stream

protected SimpleCharStream input_stream

curChar

protected char curChar

Constructor Detail

WordTokenizerTokenManager

public WordTokenizerTokenManager(SimpleCharStream stream)

Constructor.

WordTokenizerTokenManager

public WordTokenizerTokenManager(SimpleCharStream stream,
                         int lexState)

Constructor.

Method Detail

setDebugStream

public void setDebugStream(PrintStream ds)

Set debug output.

ReInit

public void ReInit(SimpleCharStream stream)

Reinitialise parser.

ReInit

public void ReInit(SimpleCharStream stream,
          int lexState)

Reinitialise parser.

SwitchTo

public void SwitchTo(int lexState)

Switch to specified lex state.

jjFillToken

protected Token jjFillToken()

getNextToken

public Token getNextToken()

Get the next Token.