Package opennlp.tools.tokenize
Class TokenSample
- java.lang.Object
-
- opennlp.tools.tokenize.TokenSample
-
- All Implemented Interfaces:
Serializable,Sample
public class TokenSample extends Object implements Sample
ATokenSampleis text with token spans.- See Also:
- Serialized Form
-
-
Field Summary
Fields Modifier and Type Field Description static StringDEFAULT_SEPARATOR_CHARS
-
Constructor Summary
Constructors Constructor Description TokenSample(String text, Span[] tokenSpans)Initializes ainstance.TokenSample(Detokenizer detokenizer, String[] tokens)Initializes ainstancevia aDetokenizer.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description booleanequals(Object obj)StringgetText()Span[]getTokenSpans()inthashCode()static TokenSampleparse(String sampleString, String separatorChars)Parses a string sample.StringtoString()
-
-
-
Field Detail
-
DEFAULT_SEPARATOR_CHARS
public static final String DEFAULT_SEPARATOR_CHARS
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
TokenSample
public TokenSample(String text, Span[] tokenSpans)
Initializes ainstance.- Parameters:
text- The text which contains the tokens. Must not benull.tokenSpans- The spans which mark the start and end of the tokens. Must not benull.
-
TokenSample
public TokenSample(Detokenizer detokenizer, String[] tokens)
Initializes ainstancevia aDetokenizer.- Parameters:
detokenizer- The text which contains the tokens. Must not benull.tokens- The tokens to be processed. Must not benull.
-
-
Method Detail
-
getText
public String getText()
- Returns:
- Retrieves the text.
-
parse
public static TokenSample parse(String sampleString, String separatorChars)
Parses a string sample.- Parameters:
sampleString- The sample to be parsed. Must not benull.separatorChars- The characters to be considered separators. SeeDEFAULT_SEPARATOR_CHARS. Must not benull.- Returns:
- A valid
TokenSampleinstance.
-
-