net.sf.okapi.steps.tokenization.engine

Class TokenScanner

java.lang.Object

net.sf.okapi.lib.extra.Component

net.sf.okapi.steps.tokenization.common.AbstractLexer

net.sf.okapi.steps.tokenization.engine.TokenScanner

All Implemented Interfaces:

Iterator<Lexem>, net.sf.okapi.lib.extra.IComponent, net.sf.okapi.lib.extra.INotifiable, ILexer

public class TokenScanner
extends AbstractLexer

Constructor Summary

Constructors

Constructor and Description
TokenScanner()

Method Summary

Modifier and Type	Method and Description
protected Class<? extends LexerRules>	lexer_getRulesClass()
protected boolean	lexer_hasNext()
protected void	lexer_init()
protected Lexem	lexer_next()
protected void	lexer_open(String text, LocaleId language, Tokens tokens)
Lexems	process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.

Methods inherited from class net.sf.okapi.steps.tokenization.common.AbstractLexer

cancel, checkRule, checkRule, close, getRules, hasNext, init, next, open, setRules

Methods inherited from class net.sf.okapi.lib.extra.Component

exec, getDescription, getName, setDescription, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Constructor Detail

TokenScanner

public TokenScanner()

Method Detail

lexer_getRulesClass

protected Class<? extends LexerRules> lexer_getRulesClass()

Overrides:

lexer_getRulesClass in class AbstractLexer

lexer_hasNext

protected boolean lexer_hasNext()

Specified by:

lexer_hasNext in class AbstractLexer

lexer_init

protected void lexer_init()

Specified by:

lexer_init in class AbstractLexer

lexer_next

protected Lexem lexer_next()

Specified by:

lexer_next in class AbstractLexer

lexer_open

protected void lexer_open(String text,
                          LocaleId language,
                          Tokens tokens)

Specified by:

lexer_open in class AbstractLexer

process

public Lexems process(String text,
                      LocaleId language,
                      Tokens tokens)

Description copied from interface: ILexer

Alternative non-iterator way of extracting lexems. In opposite to open()-hasNext()-next()-close(), all extraction is done by a single method call. Implementations might be less memory-effective compared to the iterator.

Parameters:

text - The string to be processed

language - The language of the text

tokens - The string to be processed

Returns:

A list of extracted lexems