com.sharethis.textrank

Class LanguageModel

java.lang.Object

com.sharethis.textrank.LanguageModel

Direct Known Subclasses:

CoreNlpLanguageModel

public abstract class LanguageModel
extends Object

Facade for handling language-specific tools in natural language processing.

Author:

paco@sharethis.com

Field Summary

Fields

Modifier and Type	Field and Description
static int	TOKEN_LENGTH_LIMIT Public definitions.

Constructor Summary

Constructors

Constructor and Description
LanguageModel()

Method Summary

Modifier and Type	Method and Description
static LanguageModel	buildLanguage(String lang_code) Factory method, loads libraries for StanfordCoreNLP based on the given language code.
abstract String	getNodeKey(String text, String pos) Prepare a stable key for a graph node (stemmed, lemmatized) from a token.
abstract boolean	isAdjective(String pos) Determine whether the given PoS tag is an adjective.
abstract boolean	isNoun(String pos) Determine whether the given PoS tag is a noun.
boolean	isRelevant(String pos) Determine whether the given PoS tag is relevant to add to the graph.
abstract Sentence	parseSentence(String text) Tokenize and tag the sentence text.
String	scrubToken(String token_text) Clean the text for a token.
abstract String[]	splitParagraph(String text) Split sentences within the paragraph text.
abstract String	stemToken(String token) Perform stemming on the given token.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOKEN_LENGTH_LIMIT

public static final int TOKEN_LENGTH_LIMIT

Public definitions.

See Also:

Constant Field Values

Constructor Detail

LanguageModel

public LanguageModel()

Method Detail

buildLanguage

public static LanguageModel buildLanguage(String lang_code)

Factory method, loads libraries for StanfordCoreNLP based on the given language code.

splitParagraph

public abstract String[] splitParagraph(String text)

Split sentences within the paragraph text.

parseSentence

public abstract Sentence parseSentence(String text)

Tokenize and tag the sentence text.

getNodeKey

public abstract String getNodeKey(String text,
                                  String pos)

Prepare a stable key for a graph node (stemmed, lemmatized) from a token.

isRelevant

public boolean isRelevant(String pos)

Determine whether the given PoS tag is relevant to add to the graph.

isNoun

public abstract boolean isNoun(String pos)

Determine whether the given PoS tag is a noun.

isAdjective

public abstract boolean isAdjective(String pos)

Determine whether the given PoS tag is an adjective.

stemToken

public abstract String stemToken(String token)

Perform stemming on the given token.

scrubToken

public String scrubToken(String token_text)

Clean the text for a token.

Parameters:

token_text - input text for a token.

Returns:

clean text