public final class TokenizerModel extends BaseModel
TokenizerModel is the model used
by a learnable Tokenizer.TokenizerME,
Serialized FormTRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY| Constructor and Description |
|---|
TokenizerModel(File modelFile)
Initializes the current instance.
|
TokenizerModel(InputStream in)
Initializes the current instance.
|
TokenizerModel(MaxentModel tokenizerModel,
Map<String,String> manifestInfoEntries,
TokenizerFactory tokenizerFactory)
Initializes the current instance.
|
TokenizerModel(URL modelURL)
Initializes the current instance.
|
| Modifier and Type | Method and Description |
|---|---|
Dictionary |
getAbbreviations() |
TokenizerFactory |
getFactory() |
MaxentModel |
getMaxentModel() |
boolean |
useAlphaNumericOptimization() |
getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serializepublic TokenizerModel(MaxentModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)
tokenizerModel - the modelmanifestInfoEntries - the manifesttokenizerFactory - the factorypublic TokenizerModel(InputStream in) throws IOException
in - the Input Stream to load the model fromIOException - if reading from the stream fails in anywayInvalidFormatException - if the stream doesn't have the expected formatpublic TokenizerModel(File modelFile) throws IOException
modelFile - the file containing the tokenizer modelIOException - if reading from the stream fails in anywaypublic TokenizerModel(URL modelURL) throws IOException
modelURL - the URL pointing to the tokenizer modelIOException - if reading from the stream fails in anywaypublic TokenizerFactory getFactory()
public MaxentModel getMaxentModel()
public Dictionary getAbbreviations()
public boolean useAlphaNumericOptimization()
Copyright © 2017 The Apache Software Foundation. All rights reserved.