public class TokenizerME extends Object
This tokenizer needs a statistical model to tokenize a text which reproduces
the tokenization observed in the training data used to create the model.
The TokenizerModel class encapsulates the model and provides
methods to create it from the binary representation.
A tokenizer instance is not thread safe. For each thread one tokenizer
must be instantiated which can share one TokenizerModel instance
to safe memory.
To train a new model {train(ObjectStream, TokenizerFactory, TrainingParameters) method
can be used.
Sample usage:
InputStream modelIn;
...
TokenizerModel model = TokenizerModel(modelIn);
Tokenizer tokenizer = new TokenizerME(model);
String tokens[] = tokenizer.tokenize("A sentence to be tokenized.");
Tokenizer,
TokenizerModel,
TokenSample| Modifier and Type | Field and Description |
|---|---|
static Pattern |
alphaNumeric
Deprecated.
As of release 1.5.2, replaced by
Factory.getAlphanumeric(String) |
static String |
NO_SPLIT
Constant indicates no token split.
|
static String |
SPLIT
Constant indicates a token split.
|
| Constructor and Description |
|---|
TokenizerME(TokenizerModel model) |
TokenizerME(TokenizerModel model,
Factory factory)
Deprecated.
use
TokenizerFactory to extend the Tokenizer
functionality |
| Modifier and Type | Method and Description |
|---|---|
double[] |
getTokenProbabilities()
Returns the probabilities associated with the most recent
calls to
AbstractTokenizer.tokenize(String) or tokenizePos(String). |
String[] |
tokenize(String s)
Splits a string into its atomic parts
|
Span[] |
tokenizePos(String d)
Tokenizes the string.
|
static TokenizerModel |
train(ObjectStream<TokenSample> samples,
TokenizerFactory factory,
TrainingParameters mlParams)
Trains a model for the
TokenizerME. |
boolean |
useAlphaNumericOptimization()
Returns the value of the alpha-numeric optimization flag.
|
public static final String SPLIT
public static final String NO_SPLIT
@Deprecated public static final Pattern alphaNumeric
Factory.getAlphanumeric(String)public TokenizerME(TokenizerModel model)
public TokenizerME(TokenizerModel model, Factory factory)
TokenizerFactory to extend the Tokenizer
functionalitypublic double[] getTokenProbabilities()
AbstractTokenizer.tokenize(String) or tokenizePos(String).public Span[] tokenizePos(String d)
d - The string to be tokenized.public static TokenizerModel train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams) throws IOException
TokenizerME.samples - the samples used for the training.factory - a TokenizerFactory to get resources frommlParams - the machine learning train parametersTokenizerModelIOException - it throws an IOException if an IOException is
thrown during IO operations on a temp file which is created
during training. Or if reading from the ObjectStream
fails.public boolean useAlphaNumericOptimization()
Copyright © 2017 The Apache Software Foundation. All rights reserved.