smile.nlp.keyword

Class CooccurrenceKeywordExtractor

java.lang.Object

smile.nlp.keyword.CooccurrenceKeywordExtractor

public class CooccurrenceKeywordExtractor
extends java.lang.Object

Keyword extraction from a single document using word co-occurrence statistical information. The algorithm was proposed by Y. Matsuo and M. Ishizuka. It consists of six steps:

1. Stem words by Porter algorithm and extract phrases based APRIORI algorithm (upto 4 words with frequency more than 3 times). Discard stop words.
2. Select the top frequent terms up to 30% of running terms.
3. Clustering frequent terms. Two terms are in the same cluster if either their Jensen-Shannon divergence or mutual information is above the threshold (0.95 * log 2, and log 2, respectively).
4. Calculate the expected co-occurrence probability
5. Calculate the refined χ2 values that removes the maximal term.
6. Output a given number of terms of largest refined χ2 values.

Constructor Summary

Constructors

Constructor and Description
CooccurrenceKeywordExtractor()

Method Summary

Modifier and Type	Method and Description
java.util.ArrayList<NGram>	extract(java.lang.String text) Returns the top 10 keywords.
java.util.ArrayList<NGram>	extract(java.lang.String text, int maxNumKeywords) Returns a given number of top keywords.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CooccurrenceKeywordExtractor

public CooccurrenceKeywordExtractor()

Method Detail

extract

public java.util.ArrayList<NGram> extract(java.lang.String text)

Returns the top 10 keywords.

Parameters:

text - A single document.

Returns:

The top 10 keywords.

extract

public java.util.ArrayList<NGram> extract(java.lang.String text,
                                          int maxNumKeywords)

Returns a given number of top keywords.

Parameters:

text - A single document.

Returns:

The top keywords.