net.sf.okapi.steps.tokenization.engine

Class Remover

java.lang.Object

net.sf.okapi.lib.extra.Component

net.sf.okapi.steps.tokenization.common.AbstractLexer

net.sf.okapi.steps.tokenization.engine.Remover

All Implemented Interfaces:

Iterator<Lexem>, net.sf.okapi.lib.extra.IComponent, net.sf.okapi.lib.extra.INotifiable, ILexer

public class Remover
extends AbstractLexer

Constructor Summary

Constructors

Constructor and Description
Remover()

Method Summary

Modifier and Type	Method and Description
boolean	lexer_hasNext()
void	lexer_init()
Lexem	lexer_next()
void	lexer_open(String text, LocaleId language, Tokens tokens)
Lexems	process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.

Methods inherited from class net.sf.okapi.steps.tokenization.common.AbstractLexer

cancel, checkRule, checkRule, close, getRules, hasNext, init, lexer_getRulesClass, next, open, setRules

Methods inherited from class net.sf.okapi.lib.extra.Component

exec, getDescription, getName, setDescription, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Constructor Detail

Remover

public Remover()

Method Detail

lexer_hasNext

public boolean lexer_hasNext()

Specified by:

lexer_hasNext in class AbstractLexer

lexer_init

public void lexer_init()

Specified by:

lexer_init in class AbstractLexer

lexer_next

public Lexem lexer_next()

Specified by:

lexer_next in class AbstractLexer

lexer_open

public void lexer_open(String text,
                       LocaleId language,
                       Tokens tokens)

Specified by:

lexer_open in class AbstractLexer

process

public Lexems process(String text,
                      LocaleId language,
                      Tokens tokens)

Description copied from interface: ILexer

Alternative non-iterator way of extracting lexems. In opposite to open()-hasNext()-next()-close(), all extraction is done by a single method call. Implementations might be less memory-effective compared to the iterator.

Parameters:

text - The string to be processed

language - The language of the text

tokens - The string to be processed

Returns:

A list of extracted lexems