net.sf.okapi.steps.tokenization.common

Class AbstractLexer

java.lang.Object

net.sf.okapi.lib.extra.Component

net.sf.okapi.steps.tokenization.common.AbstractLexer

All Implemented Interfaces:

Iterator<Lexem>, net.sf.okapi.lib.extra.IComponent, net.sf.okapi.lib.extra.INotifiable, ILexer

Direct Known Subclasses:

GarbageCollector, JavaCcLexer, LookUpLexer, RbbiLexer, Reconciler, Remover, Retainer, Sorter, TextScanner, TokenMatcher, TokenScanner

public abstract class AbstractLexer
extends net.sf.okapi.lib.extra.Component
implements ILexer

Constructor Summary

Constructors

Constructor and Description
AbstractLexer()

Method Summary

Modifier and Type	Method and Description
void	cancel() Cancels the current process.
protected boolean	checkRule(LexerRule rule)
protected boolean	checkRule(LexerRule rule, LocaleId language)
void	close() Called after the lexer is done with extracting lexems.
LexerRules	getRules() Gets the current rules for this lexer.
boolean	hasNext() Indicates if there is a lexem extracted.
void	init() Initializes the lexer.
protected Class<? extends LexerRules>	lexer_getRulesClass()
protected abstract boolean	lexer_hasNext()
protected abstract void	lexer_init()
protected abstract Lexem	lexer_next()
protected abstract void	lexer_open(String text, LocaleId language, Tokens tokens)
Lexem	next() Gets the next lexem available.
void	open(String text, LocaleId language, Tokens tokens) Starts processing a string or a list of tokens, extracting lexems from them.
void	setRules(LexerRules rules) Sets new rules for this lexer.

Methods inherited from class net.sf.okapi.lib.extra.Component

exec, getDescription, getName, setDescription, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface net.sf.okapi.steps.tokenization.common.ILexer

process

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Constructor Detail

AbstractLexer

public AbstractLexer()

Method Detail

lexer_init

protected abstract void lexer_init()

lexer_hasNext

protected abstract boolean lexer_hasNext()

lexer_next

protected abstract Lexem lexer_next()

lexer_open

protected abstract void lexer_open(String text,
                                   LocaleId language,
                                   Tokens tokens)

init

public void init()

Description copied from interface: ILexer

Initializes the lexer.

Specified by:

init in interface ILexer

cancel

public void cancel()

Description copied from interface: ILexer

Cancels the current process.

Specified by:

cancel in interface ILexer

close

public void close()

Description copied from interface: ILexer

Called after the lexer is done with extracting lexems.

Specified by:

close in interface ILexer

getRules

public LexerRules getRules()

Description copied from interface: ILexer

Gets the current rules for this lexer.

Specified by:

getRules in interface ILexer

Returns:

The current rules for this lexer

hasNext

public boolean hasNext()

Description copied from interface: ILexer

Indicates if there is a lexem extracted.

Specified by:

hasNext in interface Iterator<Lexem>

Specified by:

hasNext in interface ILexer

Returns:

True if there is at least one lexem has been extracted, false if none

next

public Lexem next()

Description copied from interface: ILexer

Gets the next lexem available.

Specified by:

next in interface Iterator<Lexem>

Specified by:

next in interface ILexer

Returns:

The next lexem available or null if there are no events

open

public void open(String text,
                 LocaleId language,
                 Tokens tokens)

Description copied from interface: ILexer

Starts processing a string or a list of tokens, extracting lexems from them.

Specified by:

open in interface ILexer

Parameters:

text - The string to be processed

language - The language of the text

tokens - The string to be processed

setRules

public void setRules(LexerRules rules)

Description copied from interface: ILexer

Sets new rules for this lexer.

Specified by:

setRules in interface ILexer

Parameters:

rules - The new rules to use

checkRule

protected boolean checkRule(LexerRule rule,
                            LocaleId language)

checkRule

protected boolean checkRule(LexerRule rule)

lexer_getRulesClass

protected Class<? extends LexerRules> lexer_getRulesClass()