ILexer (Okapi Step for Tokenization 1.39.0 API)

All Superinterfaces:

Iterator<Lexem>

All Known Implementing Classes:

AbstractLexer, GarbageCollector, JavaCcLexer, LookUpLexer, RbbiLexer, Reconciler, Remover, Retainer, Sorter, TextScanner, TokenMatcher, TokenScanner
```
public interface ILexer
extends Iterator<Lexem>
```
Common set of methods to extract lexems from a string or a list of tokens.
There are 2 ways of extraction provided: iterator and a single call. The implementing class can implement either of the 2 methods. The iterator way is normally more memory-effective, though the single call can be desirable when an existing lexer/tokenizer object is being wrapped around by this interface.
Lexers, implementing this interface, can do the following things:
split up a string into lexems
split up longer lexems on a token list, providing further tokenization of existing tokens

merge shorter lexems on a token list, providing further tokenization of existing tokens.

Example of iterator:

 ILexer lexer; 
 Tokens tokens = new Tokens();
 
 lexer.open("Tokenize me!", tokens);
 
 while (lexer.hasNext())
     Lexem lexem = lexer.next();
     
 lexer.close;

Example of a single call:

 ILexer lexer; 
 Tokens tokens = new Tokens();
 
 Lexems lexems = lexer.process("Tokenize me!", tokens);

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method and Description
`void`	`cancel()` Cancels the current process.
`void`	`close()` Called after the lexer is done with extracting lexems.
`LexerRules`	`getRules()` Gets the current rules for this lexer.
`boolean`	`hasNext()` Indicates if there is a lexem extracted.
`void`	`init()` Initializes the lexer.
`Lexem`	`next()` Gets the next lexem available.
`void`	`open(String text, LocaleId language, Tokens tokens)` Starts processing a string or a list of tokens, extracting lexems from them.
`Lexems`	`process(String text, LocaleId language, Tokens tokens)` Alternative non-iterator way of extracting lexems.
`void`	`setRules(LexerRules rules)` Sets new rules for this lexer.

Methods inherited from interface java.util.Iterator
forEachRemaining, remove

- Method Detail
  - init
```
void init()
```
    Initializes the lexer.
  - getRules
```
LexerRules getRules()
```
    Gets the current rules for this lexer.
    
    Returns:
    
    The current rules for this lexer
  - setRules
```
void setRules(LexerRules rules)
```
    Sets new rules for this lexer.
    
    Parameters:
    
    rules - The new rules to use
  - open
```
void open(String text,
          LocaleId language,
          Tokens tokens)
```
    Starts processing a string or a list of tokens, extracting lexems from them.
    
    Parameters:
    
    text - The string to be processed
    
    language - The language of the text
    
    tokens - The string to be processed
  - process
```
Lexems process(String text,
               LocaleId language,
               Tokens tokens)
```
    Alternative non-iterator way of extracting lexems. In opposite to open()-hasNext()-next()-close(), all extraction is done by a single method call. Implementations might be less memory-effective compared to the iterator.
    
    Parameters:
    
    text - The string to be processed
    
    language - The language of the text
    
    tokens - The string to be processed
    
    Returns:
    
    A list of extracted lexems
  - hasNext
```
boolean hasNext()
```
    Indicates if there is a lexem extracted.
    
    Specified by:
    
    hasNext in interface Iterator<Lexem>
    
    Returns:
    
    True if there is at least one lexem has been extracted, false if none
  - next
```
Lexem next()
```
    Gets the next lexem available.
    
    Specified by:
    
    next in interface Iterator<Lexem>
    
    Returns:
    
    The next lexem available or null if there are no events
  - close
```
void close()
```
    Called after the lexer is done with extracting lexems.
  - cancel
```
void cancel()
```
    Cancels the current process.

Interface ILexer

Method Summary

Methods inherited from interface java.util.Iterator

Method Detail

init

getRules

setRules

open

process

hasNext

next

close

cancel