net.sf.okapi.steps.tokenization.engine

Class Retainer

java.lang.Object

net.sf.okapi.lib.extra.Component

net.sf.okapi.steps.tokenization.common.AbstractLexer

net.sf.okapi.steps.tokenization.engine.Retainer

All Implemented Interfaces:

net.sf.okapi.lib.extra.IComponent, net.sf.okapi.lib.extra.INotifiable, ILexer

public class Retainer
extends AbstractLexer

Constructor Summary

Constructors

Constructor and Description
Retainer()

Method Summary

Methods

Modifier and Type	Method and Description
boolean	lexer_hasNext()
void	lexer_init()
Lexem	lexer_next()
void	lexer_open(String text, LocaleId language, Tokens tokens)
Lexems	process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.

Methods inherited from class net.sf.okapi.steps.tokenization.common.AbstractLexer

cancel, checkRule, checkRule, close, getRules, hasNext, init, lexer_getRulesClass, next, open, setRules

Methods inherited from class net.sf.okapi.lib.extra.Component

exec, getDescription, getName, setDescription, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Retainer

public Retainer()

Method Detail

lexer_hasNext

public boolean lexer_hasNext()

Specified by:

lexer_hasNext in class AbstractLexer

lexer_init

public void lexer_init()

Specified by:

lexer_init in class AbstractLexer

lexer_next

public Lexem lexer_next()

Specified by:

lexer_next in class AbstractLexer

lexer_open

public void lexer_open(String text,
              LocaleId language,
              Tokens tokens)

Specified by:

lexer_open in class AbstractLexer

process

public Lexems process(String text,
             LocaleId language,
             Tokens tokens)

Description copied from interface: ILexer

Alternative non-iterator way of extracting lexems. In opposite to open()-hasNext()-next()-close(), all extraction is done by a single method call. Implementations might be less memory-effective compared to the iterator.

Parameters:

text - The string to be processed

language - The language of the text

tokens - The string to be processed

Returns:

A list of extracted lexems