A B C D E F G H I J K L M N O P R S T U W

A

AbstractLexer - Class in net.sf.okapi.steps.tokenization.common

AbstractLexer() - Constructor for class net.sf.okapi.steps.tokenization.common.AbstractLexer

ACRONYM - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

add(LexerRule) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

add(int, LexerRule) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

addAll(Collection<? extends LexerRule>) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

addAll(int, Collection<? extends LexerRule>) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

addEscapes(String) - Static method in error net.sf.okapi.steps.tokenization.engine.javacc.TokenMgrError

Replaces unprintable characters by their escaped (or unicode escaped) equivalents in the given string

addTokenItem(String, String) - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

addTokens(Tokens) - Method in class net.sf.okapi.steps.tokenization.common.TokensAnnotation

adjustBeginLineColumn(int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Method to adjust line and column numbers for the start of a token.

ALPHA - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

ALPHANUM - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

APOSTROPHE - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

B

backup(int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Backup a number of characters.

beginColumn - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

The column number of the first character of this Token.

beginLine - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

The line number of the first character of this Token.

BeginToken() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Start.

bufcolumn - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

buffer - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

bufline - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

bufpos - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Position in buffer.

C

cancel() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

cancel() - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Cancels the current process.

checkRule(LexerRule, LocaleId) - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

checkRule(LexerRule) - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

clear() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

close() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

close() - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Called after the lexer is done with extracting lexems.

column - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

COMPANY - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

component_init() - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

Config - Class in net.sf.okapi.steps.tokenization.common

Config() - Constructor for class net.sf.okapi.steps.tokenization.common.Config

contains(Object) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

containsAll(Collection<?>) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

curChar - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

currentToken - Variable in exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

This is the last token that has been consumed successfully.

D

debugStream - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Debug output.

DEFAULT - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

Lexical state.

delete() - Method in class net.sf.okapi.steps.tokenization.common.Token

description - Variable in class net.sf.okapi.steps.tokenization.common.StructureParameters

DIGIT - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

disable_tracing() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Disable tracing.

Done() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reset buffer when finished.

E

EMAIL - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

enable_tracing() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Enable tracing.

endColumn - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

The column number of the last character of this Token.

endLine - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

The line number of the last character of this Token.

EOF - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

End of File.

eol - Variable in exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

The end of line string for this machine.

exec(Object, String, Object) - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

ExpandBuff(boolean) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

expectedTokenSequences - Variable in exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

Each entry in this array is an array of integers.

F

FillBuff() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

fixRanges(List<Integer>) - Method in class net.sf.okapi.steps.tokenization.tokens.Tokens

formatLanguageInfo(ULocale) - Static method in class net.sf.okapi.steps.tokenization.locale.LanguageList

formatRule(String, String, String, String, int) - Static method in class net.sf.okapi.steps.tokenization.engine.RbbiLexer

G

GarbageCollector - Class in net.sf.okapi.steps.tokenization.engine

GarbageCollector() - Constructor for class net.sf.okapi.steps.tokenization.engine.GarbageCollector

generateParseException() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Generate ParseException.

get(int) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

getAllLanguages() - Static method in class net.sf.okapi.steps.tokenization.locale.LanguageList

getAnnotation(Class<A>) - Method in class net.sf.okapi.steps.tokenization.common.Lexem

getAnnotations() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

getBeginColumn() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Get token beginning column number.

getBeginLine() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Get token beginning line number.

getColumn() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Deprecated.

getConfigInfo() - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

getDescription() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getDescription() - Method in class net.sf.okapi.steps.tokenization.common.Token

getDescription() - Method in class net.sf.okapi.steps.tokenization.tokens.TokenItem

getDisplayName(String) - Static method in class net.sf.okapi.steps.tokenization.locale.LanguageList

getEndColumn() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Get token end column number.

getEndLine() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Get token end line number.

getEngineConfig() - Method in class net.sf.okapi.steps.tokenization.common.Config

getFilteredList(String...) - Method in class net.sf.okapi.steps.tokenization.common.TokensAnnotation

getFilteredList(String...) - Method in class net.sf.okapi.steps.tokenization.tokens.Tokens

Return a list of Token objects.

getId() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

Gets lexem ID.

getIdleRules() - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

GetImage() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Get token literal value.

getInputToken() - Method in class net.sf.okapi.steps.tokenization.common.InputTokenAnnotation

getInTokenIDs() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getItems() - Method in class net.sf.okapi.steps.tokenization.common.StructureParameters

getItems() - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

getKeepInput() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getLanguageCodes_ICU() - Static method in class net.sf.okapi.steps.tokenization.locale.LanguageList

getLanguageCodes_Okapi() - Static method in class net.sf.okapi.steps.tokenization.locale.LanguageList

getLanguages() - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

getLanguages() - Static method in class net.sf.okapi.steps.tokenization.locale.LanguageList

getLexem() - Method in class net.sf.okapi.steps.tokenization.common.Token

getLexemId() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getLexemId() - Method in class net.sf.okapi.steps.tokenization.common.Token

getLexerClass() - Method in class net.sf.okapi.steps.tokenization.common.StructureParametersItem

getLexerId() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

Gets ID of the lexer that extracted this lexem.

getLexerId() - Method in class net.sf.okapi.steps.tokenization.common.Token

getLexers() - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

getLine() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Deprecated.

getLocaleFilter() - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

getMessage() - Method in error net.sf.okapi.steps.tokenization.engine.javacc.TokenMgrError

You can also modify the body of this method to customize your error messages.

getName() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getName() - Method in class net.sf.okapi.steps.tokenization.common.Token

getName() - Method in class net.sf.okapi.steps.tokenization.tokens.TokenItem

getNextToken() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Get the next Token.

getNextToken() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Get the next Token.

getOutTokenIDs() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getPattern() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getRange() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

getRange() - Method in class net.sf.okapi.steps.tokenization.common.Token

getRegexGroup() - Method in class net.sf.okapi.steps.tokenization.common.RegexRule

getRegexOptions() - Method in class net.sf.okapi.steps.tokenization.common.RegexRule

getRule(int) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

Gets a lexer rule for the given lexem ID.

getRuleClass() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

getRuleClass() - Method in class net.sf.okapi.steps.tokenization.common.RegexRules

getRules() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

getRules() - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Gets the current rules for this lexer.

getRulesLocation() - Method in class net.sf.okapi.steps.tokenization.common.StructureParametersItem

getSample() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getScore() - Method in class net.sf.okapi.steps.tokenization.common.Token

GetSuffix(int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Get the suffix.

getTabSize(int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

getToken(int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Get the specific Token.

getTokenDescription(int) - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

getTokenDescription(String) - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

getTokenId() - Method in class net.sf.okapi.steps.tokenization.common.Token

Gets integer identifier of the token.

getTokenId(String) - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

getTokenIDs() - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

Gets list of IDs of all tokens.

getTokenIDs(List<String>) - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

Gets list of IDs of the given tokens.

getTokenItems() - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

getTokenName(int) - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

getTokenNames() - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

getTokenNames() - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

Gets list of names of all tokens.

getTokenNamesStr() - Static method in class net.sf.okapi.steps.tokenization.tokens.Tokens

getTokens() - Method in class net.sf.okapi.steps.tokenization.common.TokensAnnotation

getUserTokenIDs() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

getValue() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

getValue() - Method in class net.sf.okapi.steps.tokenization.common.Token

getValue() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.Token

An optional attribute value of the Token.

H

handleTextUnit(Event) - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

HAS_DIGIT - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

hasNext() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

hasNext() - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Indicates if there is a lexem extracted.

hasOutTokens() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

HOST - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

HYPHENATED - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

I

ILexer - Interface in net.sf.okapi.steps.tokenization.common

Common set of methods to extract lexems from a string or a list of tokens.

image - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

The string image of the token.

inBuf - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

indexOf(Object) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

init() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

init() - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Initializes the lexer.

input_stream - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

inputStream - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

InputTokenAnnotation - Class in net.sf.okapi.steps.tokenization.common

InputTokenAnnotation(Token) - Constructor for class net.sf.okapi.steps.tokenization.common.InputTokenAnnotation

isDeleted() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

isDeleted() - Method in class net.sf.okapi.steps.tokenization.common.Token

isEmpty() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

isEnabled() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

isEnabled() - Method in class net.sf.okapi.steps.tokenization.common.StructureParametersItem

isImmutable() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

isImmutable() - Method in class net.sf.okapi.steps.tokenization.common.Token

iterator() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

J

JavaCcLexer - Class in net.sf.okapi.steps.tokenization.engine

JavaCcLexer() - Constructor for class net.sf.okapi.steps.tokenization.engine.JavaCcLexer

jj_nt - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Next token.

jjFillToken() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

jjstrLiteralImages - Static variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Token literal values.

K

kind - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

An integer that describes the kind of this token.

L

LanguageAndTokenParameters - Class in net.sf.okapi.steps.tokenization.common

LanguageAndTokenParameters() - Constructor for class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

LanguageList - Class in net.sf.okapi.steps.tokenization.locale

LanguageList() - Constructor for class net.sf.okapi.steps.tokenization.locale.LanguageList

LanguageParameters - Class in net.sf.okapi.steps.tokenization.common

LanguageParameters() - Constructor for class net.sf.okapi.steps.tokenization.common.LanguageParameters

lastIndexOf(Object) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

LETTER - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

Lexem - Class in net.sf.okapi.steps.tokenization.common

Lexem(int, String, Range) - Constructor for class net.sf.okapi.steps.tokenization.common.Lexem

Lexem(int, String, int, int) - Constructor for class net.sf.okapi.steps.tokenization.common.Lexem

Lexems - Class in net.sf.okapi.steps.tokenization.common

Lexems() - Constructor for class net.sf.okapi.steps.tokenization.common.Lexems

lexer_getRulesClass() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

lexer_getRulesClass() - Method in class net.sf.okapi.steps.tokenization.engine.TextScanner

lexer_getRulesClass() - Method in class net.sf.okapi.steps.tokenization.engine.TokenMatcher

lexer_getRulesClass() - Method in class net.sf.okapi.steps.tokenization.engine.TokenScanner

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.GarbageCollector

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.JavaCcLexer

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.LookUpLexer

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.RbbiLexer

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.Reconciler

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.Remover

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.Retainer

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.Sorter

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.TextScanner

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.TokenMatcher

lexer_hasNext() - Method in class net.sf.okapi.steps.tokenization.engine.TokenScanner

lexer_init() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.GarbageCollector

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.JavaCcLexer

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.LookUpLexer

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.RbbiLexer

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.Reconciler

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.Remover

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.Retainer

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.Sorter

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.TextScanner

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.TokenMatcher

lexer_init() - Method in class net.sf.okapi.steps.tokenization.engine.TokenScanner

lexer_next() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.GarbageCollector

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.JavaCcLexer

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.LookUpLexer

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.RbbiLexer

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.Reconciler

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.Remover

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.Retainer

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.Sorter

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.TextScanner

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.TokenMatcher

lexer_next() - Method in class net.sf.okapi.steps.tokenization.engine.TokenScanner

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.GarbageCollector

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.JavaCcLexer

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.LookUpLexer

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.RbbiLexer

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Reconciler

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Remover

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Retainer

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Sorter

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.TextScanner

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.TokenMatcher

lexer_open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.TokenScanner

LexerRule - Class in net.sf.okapi.steps.tokenization.common

LexerRule() - Constructor for class net.sf.okapi.steps.tokenization.common.LexerRule

LexerRules - Class in net.sf.okapi.steps.tokenization.common

Lexer rules

LexerRules() - Constructor for class net.sf.okapi.steps.tokenization.common.LexerRules

LexicalError(boolean, int, int, int, String, char) - Static method in error net.sf.okapi.steps.tokenization.engine.javacc.TokenMgrError

Returns a detailed message for the Error when it is thrown by the token manager to indicate a lexical error.

lexStateNames - Static variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Lexer state names.

line - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

listIterator() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

listIterator(int) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

loadItems() - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

LocaleUtil - Class in net.sf.okapi.steps.tokenization.locale

LocaleUtil() - Constructor for class net.sf.okapi.steps.tokenization.locale.LocaleUtil

LookUpLexer - Class in net.sf.okapi.steps.tokenization.engine

LookUpLexer() - Constructor for class net.sf.okapi.steps.tokenization.engine.LookUpLexer

M

maxNextCharInd - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

N

net.sf.okapi.steps.tokenization - package net.sf.okapi.steps.tokenization

net.sf.okapi.steps.tokenization.common - package net.sf.okapi.steps.tokenization.common

net.sf.okapi.steps.tokenization.engine - package net.sf.okapi.steps.tokenization.engine

net.sf.okapi.steps.tokenization.engine.javacc - package net.sf.okapi.steps.tokenization.engine.javacc

net.sf.okapi.steps.tokenization.locale - package net.sf.okapi.steps.tokenization.locale

net.sf.okapi.steps.tokenization.tokens - package net.sf.okapi.steps.tokenization.tokens

newToken(int, String) - Static method in class net.sf.okapi.steps.tokenization.engine.javacc.Token

Returns a new Token object, by default.

newToken(int) - Static method in class net.sf.okapi.steps.tokenization.engine.javacc.Token

next() - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

next() - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Gets the next lexem available.

next - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

A reference to the next regular (non-special) token from the input stream.

nextToken() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Returns the next token in the stream, or null at EOS.

NOISE - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

normalizeLanguageCode_ICU(String) - Static method in class net.sf.okapi.steps.tokenization.locale.LocaleUtil

Converts the language tag (language ID and an optional region/country part) to the ICU format: lower case for the language, upper case for the region, parts are delimited with an underscore.

normalizeLanguageCode_Okapi(String) - Static method in class net.sf.okapi.steps.tokenization.locale.LocaleUtil

Deprecated.

don't use this. It only "understands" language + region, but that is just a small part of what BCP 47 supports. Also, the LocaleId does case normalization out of the box now.

normalizeLanguageCodes_Okapi(List<String>) - Static method in class net.sf.okapi.steps.tokenization.locale.LocaleUtil

Deprecated.

don't use this. It only "understands" language + region, but that is just a small part of what BCP 47 supports. Also, the LocaleId does case normalization out of the box now.

NUM - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

O

open(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

open(String, LocaleId, Tokens) - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Starts processing a string or a list of tokens, extracting lexems from them.

P

P - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

RegularExpression Id.

Parameters - Class in net.sf.okapi.steps.tokenization

Tokenization step parameters

Parameters() - Constructor for class net.sf.okapi.steps.tokenization.Parameters

Parameters - Class in net.sf.okapi.steps.tokenization.tokens

Parameters() - Constructor for class net.sf.okapi.steps.tokenization.tokens.Parameters

parameters_init() - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

parameters_init() - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

parameters_init() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

parameters_init() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

parameters_init() - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.Config

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.RegexRule

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.StructureParameters

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.StructureParametersItem

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.Parameters

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

parameters_load(ParametersString) - Method in class net.sf.okapi.steps.tokenization.tokens.TokenItem

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.Config

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.RegexRule

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.StructureParameters

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.common.StructureParametersItem

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.Parameters

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

parameters_reset() - Method in class net.sf.okapi.steps.tokenization.tokens.TokenItem

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.Config

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.RegexRule

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.StructureParameters

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.common.StructureParametersItem

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.Parameters

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

parameters_save(ParametersString) - Method in class net.sf.okapi.steps.tokenization.tokens.TokenItem

ParseException - Exception in net.sf.okapi.steps.tokenization.engine.javacc

This exception is thrown when parse errors are encountered.

ParseException(Token, int[][], String[]) - Constructor for exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

This constructor is used by the method "generateParseException" in the generated parser.

ParseException() - Constructor for exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

The following constructors are for use by you for whatever purpose you can think of.

ParseException(String) - Constructor for exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

Constructor with message.

prevCharIsCR - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

prevCharIsLF - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

process(String, LocaleId, Tokens) - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Alternative non-iterator way of extracting lexems.

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.GarbageCollector

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.JavaCcLexer

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.LookUpLexer

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.RbbiLexer

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Reconciler

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Remover

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Retainer

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.Sorter

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.TextScanner

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.TokenMatcher

process(String, LocaleId, Tokens) - Method in class net.sf.okapi.steps.tokenization.engine.TokenScanner

R

RAWTEXT - Static variable in class net.sf.okapi.steps.tokenization.TokenizationStep

RbbiLexer - Class in net.sf.okapi.steps.tokenization.engine

RbbiLexer() - Constructor for class net.sf.okapi.steps.tokenization.engine.RbbiLexer

readChar() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Read a character.

Reconciler - Class in net.sf.okapi.steps.tokenization.engine

Reconciler() - Constructor for class net.sf.okapi.steps.tokenization.engine.Reconciler

RegexRule - Class in net.sf.okapi.steps.tokenization.common

RegexRule() - Constructor for class net.sf.okapi.steps.tokenization.common.RegexRule

RegexRules - Class in net.sf.okapi.steps.tokenization.common

RegexRules() - Constructor for class net.sf.okapi.steps.tokenization.common.RegexRules

ReInit(Reader, int, int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(Reader, int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(Reader) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream, String, int, int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream, int, int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream, String) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream, String, int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream, int, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Reinitialise.

ReInit(InputStream) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Reinitialise.

ReInit(InputStream, String) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Reinitialise.

ReInit(Reader) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Reinitialise.

ReInit(WordTokenizerTokenManager) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Reinitialise.

ReInit(SimpleCharStream) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Reinitialise parser.

ReInit(SimpleCharStream, int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Reinitialise parser.

remove(Object) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

remove(int) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

removeAll(Collection<?>) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

removeAnnotation(Class<A>) - Method in class net.sf.okapi.steps.tokenization.common.Lexem

Remover - Class in net.sf.okapi.steps.tokenization.engine

Remover() - Constructor for class net.sf.okapi.steps.tokenization.engine.Remover

retainAll(Collection<?>) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

Retainer - Class in net.sf.okapi.steps.tokenization.engine

Retainer() - Constructor for class net.sf.okapi.steps.tokenization.engine.Retainer

S

saveItems() - Method in class net.sf.okapi.steps.tokenization.tokens.Parameters

set(int, LexerRule) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

setAnnotation(IAnnotation) - Method in class net.sf.okapi.steps.tokenization.common.Lexem

setConfiguration(Class<?>, String) - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

setDebugStream(PrintStream) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Set debug output.

setDeleted(boolean) - Method in class net.sf.okapi.steps.tokenization.common.Lexem

setDeleted(boolean) - Method in class net.sf.okapi.steps.tokenization.common.Token

setDescription(String) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setEnabled(boolean) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setImmutable(boolean) - Method in class net.sf.okapi.steps.tokenization.common.Lexem

setImmutable(boolean) - Method in class net.sf.okapi.steps.tokenization.common.Token

setImmutable(boolean) - Method in class net.sf.okapi.steps.tokenization.tokens.Tokens

setKeepInput(boolean) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setLexemId(int) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setLexerId(int) - Method in class net.sf.okapi.steps.tokenization.common.Lexem

setLexers(List<ILexer>) - Method in class net.sf.okapi.steps.tokenization.TokenizationStep

setLocaleFilter(LocaleFilter) - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

setLocaleFilter(String) - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

setName(String) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setPattern(String) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setRules(LexerRules) - Method in class net.sf.okapi.steps.tokenization.common.AbstractLexer

setRules(LexerRules) - Method in interface net.sf.okapi.steps.tokenization.common.ILexer

Sets new rules for this lexer.

setSample(String) - Method in class net.sf.okapi.steps.tokenization.common.LexerRule

setScore(int) - Method in class net.sf.okapi.steps.tokenization.common.Token

setTabSize(int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

setTokenNames(String...) - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

setTokens(Tokens) - Method in class net.sf.okapi.steps.tokenization.common.TokensAnnotation

SimpleCharStream - Class in net.sf.okapi.steps.tokenization.engine.javacc

An implementation of interface CharStream, where the stream is assumed to contain only ASCII characters (without unicode processing).

SimpleCharStream(Reader, int, int, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(Reader, int, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(Reader) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(InputStream, String, int, int, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(InputStream, int, int, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(InputStream, String, int, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(InputStream, int, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(InputStream, String) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

SimpleCharStream(InputStream) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Constructor.

size() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

Sorter - Class in net.sf.okapi.steps.tokenization.engine

Sorter() - Constructor for class net.sf.okapi.steps.tokenization.engine.Sorter

specialToken - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.Token

This field is used to access special tokens that occur prior to this token, but after the immediately preceding regular (non-special) token.

staticFlag - Static variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

Whether parser is static.

StructureParameters - Class in net.sf.okapi.steps.tokenization.common

StructureParameters() - Constructor for class net.sf.okapi.steps.tokenization.common.StructureParameters

StructureParametersItem - Class in net.sf.okapi.steps.tokenization.common

StructureParametersItem() - Constructor for class net.sf.okapi.steps.tokenization.common.StructureParametersItem

subList(int, int) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

supportsLanguage(LocaleId) - Method in class net.sf.okapi.steps.tokenization.common.LanguageParameters

supportsToken(String) - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

supportsToken(int) - Method in class net.sf.okapi.steps.tokenization.common.LanguageAndTokenParameters

SwitchTo(int) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Switch to specified lex state.

T

tabSize - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

TextScanner - Class in net.sf.okapi.steps.tokenization.engine

TextScanner() - Constructor for class net.sf.okapi.steps.tokenization.engine.TextScanner

toArray() - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

toArray(T[]) - Method in class net.sf.okapi.steps.tokenization.common.LexerRules

Token - Class in net.sf.okapi.steps.tokenization.common

Token(int, Lexem, int) - Constructor for class net.sf.okapi.steps.tokenization.common.Token

Token - Class in net.sf.okapi.steps.tokenization.engine.javacc

Describes the input token stream.

Token() - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.Token

No-argument constructor

Token(int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.Token

Constructs a new token for the specified Image.

Token(int, String) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.Token

Constructs a new token for the specified Image and Kind.

token - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Current token.

token_source - Variable in class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Generated Token Manager.

tokenImage - Variable in exception net.sf.okapi.steps.tokenization.engine.javacc.ParseException

This is a reference to the "tokenImage" array of the generated parser within which the parse error occurred.

tokenImage - Static variable in interface net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerConstants

Literal token values.

TokenItem - Class in net.sf.okapi.steps.tokenization.tokens

TokenItem() - Constructor for class net.sf.okapi.steps.tokenization.tokens.TokenItem

TokenItem(String, String) - Constructor for class net.sf.okapi.steps.tokenization.tokens.TokenItem

TokenizationStep - Class in net.sf.okapi.steps.tokenization

TokenizationStep() - Constructor for class net.sf.okapi.steps.tokenization.TokenizationStep

tokenize(ITextUnit, LocaleId, String...) - Static method in class net.sf.okapi.steps.tokenization.Tokenizer

tokenize(TextContainer, LocaleId, String...) - Static method in class net.sf.okapi.steps.tokenization.Tokenizer

tokenize(TextFragment, LocaleId, String...) - Static method in class net.sf.okapi.steps.tokenization.Tokenizer

tokenize(String, LocaleId, String...) - Static method in class net.sf.okapi.steps.tokenization.Tokenizer

Tokenizer - Class in net.sf.okapi.steps.tokenization

Tokenizer() - Constructor for class net.sf.okapi.steps.tokenization.Tokenizer

tokenizeSource - Variable in class net.sf.okapi.steps.tokenization.Parameters

tokenizeString(String, LocaleId, String...) - Static method in class net.sf.okapi.steps.tokenization.Tokenizer

Extracts tokens from the given text.

tokenizeTargets - Variable in class net.sf.okapi.steps.tokenization.Parameters

TokenMatcher - Class in net.sf.okapi.steps.tokenization.engine

TokenMatcher() - Constructor for class net.sf.okapi.steps.tokenization.engine.TokenMatcher

TokenMgrError - Error in net.sf.okapi.steps.tokenization.engine.javacc

Token Manager Error.

TokenMgrError() - Constructor for error net.sf.okapi.steps.tokenization.engine.javacc.TokenMgrError

No arg constructor.

TokenMgrError(String, int) - Constructor for error net.sf.okapi.steps.tokenization.engine.javacc.TokenMgrError

Constructor with message and reason.

TokenMgrError(boolean, int, int, int, String, char, int) - Constructor for error net.sf.okapi.steps.tokenization.engine.javacc.TokenMgrError

Full Constructor.

Tokens - Class in net.sf.okapi.steps.tokenization.tokens

Tokens() - Constructor for class net.sf.okapi.steps.tokenization.tokens.Tokens

TokensAnnotation - Class in net.sf.okapi.steps.tokenization.common

TokensAnnotation(Tokens) - Constructor for class net.sf.okapi.steps.tokenization.common.TokensAnnotation

TokenScanner - Class in net.sf.okapi.steps.tokenization.engine

TokenScanner() - Constructor for class net.sf.okapi.steps.tokenization.engine.TokenScanner

toString() - Method in class net.sf.okapi.steps.tokenization.common.InputTokenAnnotation

toString() - Method in class net.sf.okapi.steps.tokenization.common.Lexem

toString() - Method in class net.sf.okapi.steps.tokenization.common.Token

toString() - Method in class net.sf.okapi.steps.tokenization.common.TokensAnnotation

toString() - Method in class net.sf.okapi.steps.tokenization.engine.javacc.Token

Returns the image.

toString() - Method in class net.sf.okapi.steps.tokenization.tokens.Tokens

ts - Static variable in class net.sf.okapi.steps.tokenization.Tokenizer

U

undelete() - Method in class net.sf.okapi.steps.tokenization.common.Token

UpdateLineColumn(char) - Method in class net.sf.okapi.steps.tokenization.engine.javacc.SimpleCharStream

W

WordTokenizer - Class in net.sf.okapi.steps.tokenization.engine.javacc

A grammar-based tokenizer constructed with JavaCC.

WordTokenizer(InputStream) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Constructor with InputStream.

WordTokenizer(InputStream, String) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Constructor with InputStream and supplied encoding

WordTokenizer(Reader) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Constructor.

WordTokenizer(WordTokenizerTokenManager) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizer

Constructor with generated Token Manager.

WordTokenizerConstants - Interface in net.sf.okapi.steps.tokenization.engine.javacc

Token literal values and constants.

WordTokenizerTokenManager - Class in net.sf.okapi.steps.tokenization.engine.javacc

Token Manager.

WordTokenizerTokenManager(SimpleCharStream) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Constructor.

WordTokenizerTokenManager(SimpleCharStream, int) - Constructor for class net.sf.okapi.steps.tokenization.engine.javacc.WordTokenizerTokenManager

Constructor.

A B C D E F G H I J K L M N O P R S T U W