net.sf.okapi.steps.tokenization.engine

Class RbbiLexer

java.lang.Object

net.sf.okapi.lib.extra.Component

net.sf.okapi.steps.tokenization.common.AbstractLexer

net.sf.okapi.steps.tokenization.engine.RbbiLexer

All Implemented Interfaces:

net.sf.okapi.lib.extra.IComponent, net.sf.okapi.lib.extra.INotifiable, ILexer

public class RbbiLexer
extends AbstractLexer

Constructor Summary

Constructors

Constructor and Description
RbbiLexer()

Method Summary

Methods

Modifier and Type	Method and Description
static String	formatRule(String buffer, String name, String description, String rule, int lexemId)
boolean	lexer_hasNext()
void	lexer_init()
Lexem	lexer_next()
void	lexer_open(String text, LocaleId language, Tokens tokens)
Lexems	process(String text, LocaleId language, Tokens tokens) Alternative non-iterator way of extracting lexems.

Methods inherited from class net.sf.okapi.steps.tokenization.common.AbstractLexer

cancel, checkRule, checkRule, close, getRules, hasNext, init, lexer_getRulesClass, next, open, setRules

Methods inherited from class net.sf.okapi.lib.extra.Component

exec, getDescription, getName, setDescription, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RbbiLexer

public RbbiLexer()

Method Detail

lexer_init

public void lexer_init()

Specified by:

lexer_init in class AbstractLexer

lexer_hasNext

public boolean lexer_hasNext()

Specified by:

lexer_hasNext in class AbstractLexer

lexer_next

public Lexem lexer_next()

Specified by:

lexer_next in class AbstractLexer

formatRule

public static String formatRule(String buffer,
                String name,
                String description,
                String rule,
                int lexemId)

lexer_open

public void lexer_open(String text,
              LocaleId language,
              Tokens tokens)

Specified by:

lexer_open in class AbstractLexer

process

public Lexems process(String text,
             LocaleId language,
             Tokens tokens)

Description copied from interface: ILexer

Alternative non-iterator way of extracting lexems. In opposite to open()-hasNext()-next()-close(), all extraction is done by a single method call. Implementations might be less memory-effective compared to the iterator.

Parameters:

text - The string to be processed

language - The language of the text

tokens - The string to be processed

Returns:

A list of extracted lexems