net.sf.okapi.steps.tokenization

Class Tokens

java.lang.Object

java.util.AbstractCollection<E>

java.util.AbstractList<E>

java.util.ArrayList<Token>

net.sf.okapi.steps.tokenization.Tokens

All Implemented Interfaces:

Serializable, Cloneable, Iterable<Token>, Collection<Token>, List<Token>, RandomAccess

public class Tokens
extends ArrayList<Token>

List of Token and configuration for each token type.

See Also:

Serialized Form

Field Summary

Fields inherited from class java.util.AbstractList

modCount

Constructor Summary

Constructors

Constructor and Description
Tokens()

Method Summary

Modifier and Type	Method and Description
void	fixRanges(List<Integer> markerPositions)
Tokens	getExcludedFilteredList(String... tokenNames) Return a list of TokenType objects.
Tokens	getFilteredList(String... tokenNames) Return a list of TokenType objects.
static Collection<TokenConfigs.TokenConfig>	getRules()
static String	getTokenDescription(int tokenId)
static String	getTokenDescription(String tokenName)
static int	getTokenId(String tokenName)
static List<Integer>	getTokenIDs() Gets list of IDs of all tokens.
static List<Integer>	getTokenIDs(List<String> tokenNames) Gets list of IDs of the given tokens.
static int	getTokenIndex(String tokenName)
static String	getTokenName(int tokenId)
static List<String>	getTokenNames() Gets list of names of all tokens.
static String	getTokenNamesStr()
String	toString()

Methods inherited from class java.util.ArrayList

add, add, addAll, addAll, clear, clone, contains, ensureCapacity, forEach, get, indexOf, isEmpty, iterator, lastIndexOf, listIterator, listIterator, remove, remove, removeAll, removeIf, removeRange, replaceAll, retainAll, set, size, sort, spliterator, subList, toArray, toArray, trimToSize

Methods inherited from class java.util.AbstractList

equals, hashCode

Methods inherited from class java.util.AbstractCollection

containsAll

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.List

containsAll, equals, hashCode

Methods inherited from interface java.util.Collection

parallelStream, stream

Constructor Detail

Tokens

public Tokens()

Method Detail

getTokenIndex

public static int getTokenIndex(String tokenName)

getTokenId

public static int getTokenId(String tokenName)

getTokenName

public static String getTokenName(int tokenId)

getTokenDescription

public static String getTokenDescription(int tokenId)

getTokenDescription

public static String getTokenDescription(String tokenName)

getTokenNamesStr

public static String getTokenNamesStr()

getRules

public static Collection<TokenConfigs.TokenConfig> getRules()

getTokenNames

public static List<String> getTokenNames()

Gets list of names of all tokens.

Returns:

List of available token names.

getTokenIDs

public static List<Integer> getTokenIDs()

Gets list of IDs of all tokens.

Returns:

List of available token IDs.

getTokenIDs

public static List<Integer> getTokenIDs(List<String> tokenNames)

Gets list of IDs of the given tokens.

Returns:

List of token IDs.

getFilteredList

public Tokens getFilteredList(String... tokenNames)

Return a list of TokenType objects. If tokenNames are specified, only the tokens with those names will be placed in the resulting list. If tokenNames is omitted, the list of all tokens will be returned.

Parameters:

tokenNames - Optional array of strings with token names to include.

Returns:

List of tokens.

getExcludedFilteredList

public Tokens getExcludedFilteredList(String... tokenNames)

Return a list of TokenType objects. If tokenNames are specified, they will be excluded.

Parameters:

tokenNames - Optional array of strings of token names to exclude.

Returns:

List of tokens.

fixRanges

public void fixRanges(List<Integer> markerPositions)

toString

public String toString()

Overrides:

toString in class AbstractCollection<Token>