net.sf.okapi.steps.tokenization

Class Tokenizer

java.lang.Object

net.sf.okapi.steps.tokenization.Tokenizer

public class Tokenizer
extends Object

Version:

0.1 08.07.2009

Field Summary

Fields

Modifier and Type	Field and Description
protected static TokenizationStep	ts

Constructor Summary

Constructors

Constructor and Description
Tokenizer()

Method Summary

Modifier and Type	Method and Description
static Tokens	tokenize(ITextUnit textUnit, LocaleId language, String... tokenNames)
static Tokens	tokenize(String string, LocaleId language, String... tokenNames)
static Tokens	tokenize(TextContainer textContainer, LocaleId language, String... tokenNames)
static Tokens	tokenize(TextFragment textFragment, LocaleId language, String... tokenNames)
protected static Tokens	tokenizeString(String text, LocaleId language, String... tokenNames) Extracts tokens from the given text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ts

protected static TokenizationStep ts

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

tokenizeString

protected static Tokens tokenizeString(String text,
                                       LocaleId language,
                                       String... tokenNames)

Extracts tokens from the given text.

Parameters:

text - Text to tokenize.

language - Language of the text.

tokenNames - Optional list of token names. If omitted, all tokens will be extracted.

Returns:

A list of Token objects.

tokenize

public static Tokens tokenize(ITextUnit textUnit,
                              LocaleId language,
                              String... tokenNames)

tokenize

public static Tokens tokenize(TextContainer textContainer,
                              LocaleId language,
                              String... tokenNames)

tokenize

public static Tokens tokenize(TextFragment textFragment,
                              LocaleId language,
                              String... tokenNames)

tokenize

public static Tokens tokenize(String string,
                              LocaleId language,
                              String... tokenNames)