de.uni_leipzig.asv.utils.tokenizer

Class ImprovedWordTokenizer

java.lang.Object

de.uni_leipzig.asv.utils.tokenizer.AbstractWordTokenizerImpl

de.uni_leipzig.asv.utils.tokenizer.ImprovedWordTokenizer

public class ImprovedWordTokenizer
extends AbstractWordTokenizerImpl

Author:

development

Field Summary

Fields inherited from class de.uni_leipzig.asv.utils.tokenizer.AbstractWordTokenizerImpl

EXPLIZIT_CUT, objAbbrevList, strAbbrevListFile

Constructor Summary

Constructors

Constructor and Description
ImprovedWordTokenizer() Creates a new instance of DefaultWordTokenizer

Method Summary

Methods

Modifier and Type	Method and Description
protected int	detectSequenceOfDot(String strWord)
String	execute(String strLine)
protected boolean	isNumber(String strWord)
protected boolean	isSequenceOfDot(String strWord)
protected boolean	needsNumberSplit(String strWord)
protected boolean	needsSplit(String strWord)
protected String	processDotAsPrefix(String strWord)
protected String	processDotAsSuffix(String strWord)
protected String	processInfixApostrophe(String strWord)
String	processNumber(String strWord)
protected String	processPrefix(String strWord)
protected String	processSingleNumber(String strWord)
protected String	processSuffix(String strWord)
protected String	processSuffix2(String strWord)
void	setBoolReplaceNumbers(boolean boolReplaceNumbers)

Methods inherited from class de.uni_leipzig.asv.utils.tokenizer.AbstractWordTokenizerImpl

getWhiteSpaceBeforeDot, getWhitespacePositions

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ImprovedWordTokenizer

public ImprovedWordTokenizer()

Creates a new instance of DefaultWordTokenizer

Method Detail

execute

public String execute(String strLine)

Overrides:

execute in class AbstractWordTokenizerImpl

processPrefix

protected String processPrefix(String strWord)

processSuffix

protected String processSuffix(String strWord)

isSequenceOfDot

protected boolean isSequenceOfDot(String strWord)

processDotAsPrefix

protected String processDotAsPrefix(String strWord)

detectSequenceOfDot

protected int detectSequenceOfDot(String strWord)

processDotAsSuffix

protected String processDotAsSuffix(String strWord)

processSuffix2

protected String processSuffix2(String strWord)

needsSplit

protected boolean needsSplit(String strWord)

needsNumberSplit

protected boolean needsNumberSplit(String strWord)

isNumber

protected boolean isNumber(String strWord)

processSingleNumber

protected String processSingleNumber(String strWord)

processInfixApostrophe

protected String processInfixApostrophe(String strWord)

processNumber

public String processNumber(String strWord)

setBoolReplaceNumbers

public void setBoolReplaceNumbers(boolean boolReplaceNumbers)