Uses of Interface
opennlp.tools.tokenize.Tokenizer
-
Packages that use Tokenizer Package Description opennlp.tools.cmdline.parser opennlp.tools.formats.brat Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).opennlp.tools.formats.muc Experimental package related to theMUCcorpus format.opennlp.tools.tokenize Contains classes related to finding token or words in a string.opennlp.tools.util.featuregen This package contains classes for generating sequence features. -
-
Uses of Tokenizer in opennlp.tools.cmdline.parser
Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer Modifier and Type Method Description static Parse[]ParserTool. parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses) -
Uses of Tokenizer in opennlp.tools.formats.brat
Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer Constructor Description BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)Creates a newBratNameSampleStream.BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)Creates a newBratNameSampleStream. -
Uses of Tokenizer in opennlp.tools.formats.muc
Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer Constructor Description MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)Initializes aMucNameContentHandler.MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)Initializes aMucNameSampleStream. -
Uses of Tokenizer in opennlp.tools.tokenize
Classes in opennlp.tools.tokenize that implement Tokenizer Modifier and Type Class Description classSimpleTokenizerA basicTokenizerimplementation which performs tokenization using character classes.classTokenizerMEATokenizerfor converting raw text into separated tokens.classWhitespaceTokenizerA basicTokenizerimplementation which performs tokenization using white spaces.classWordpieceTokenizerATokenizerimplementation which performs tokenization using word pieces.Constructors in opennlp.tools.tokenize with parameters of type Tokenizer Constructor Description TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)Initializes an instance to evaluate aTokenizer.TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)Initializes ainstance. -
Uses of Tokenizer in opennlp.tools.util.featuregen
Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer Constructor Description TokenPatternFeatureGenerator(Tokenizer supportTokenizer)Initializes aTokenPatternFeatureGeneratorinstance.
-