net.seninp.jmotif.text

Class TextProcessor

java.lang.Object

net.seninp.jmotif.text.TextProcessor

public final class TextProcessor
extends Object

Implements text statistics and mining utilities.

Author:

psenin

Constructor Summary

Constructors

Constructor and Description
TextProcessor() Constructor.

Method Summary

Modifier and Type	Method and Description
double	augmentedTF(WordBag bag, String term) Compute TF (term frequency) metrics.
String	bagsToTable(List<WordBag> bags)
int	classify(String classKey, double[] series, HashMap<String,HashMap<String,Double>> tfidf, Params params)
int	classify(String trueClassLabel, WordBag test, HashMap<String,HashMap<String,Double>> tfidf)
String	classify(WordBag test, HashMap<String,HashMap<String,Double>> tfidf)
HashMap<String,HashMap<String,Double>>	computeTFIDF(Collection<WordBag> texts) Computes TF*IDF values.
double	cosineDistance(HashMap<String,Double> map1, HashMap<String,Double> map2) Computes a cosine similarity.
double	cosineSimilarity(WordBag testSample, HashMap<String,Double> weightVector)
double	cosineSimilarityInstrumented(WordBag testSample, HashMap<String,Double> weightVector, HashMap<String,Double> insight)
int	df(HashMap<String,WordBag> bags, String string) Compute document frequency, DF, metrics.
double	dotProduct(double[] vector1, double[] vector2) Compute the dot product of two vectors.
double	dotProduct(Double[] vector1, Double[] vector2) Compute the dot product of two vectors.
CosineDistanceMatrix	getCosineDistanceMatrix(HashMap<String,HashMap<String,Double>> tfidf)
double	idf(HashMap<String,WordBag> bags, String string) Compute idf (inverse document frequency) metrics.
List<WordBag>	labeledSeries2WordBags(Map<String,List<double[]>> data, Params params)
double	logAveTF(WordBag bag, String term) Compute TF (term frequency) metrics.
double	logTF(WordBag bag, String term) Compute TF (term frequency) metrics.
double	magnitude(double[] vector) Compute the magnitude of the vector.
double	magnitude(Double[] vector) Compute the magnitude of the vector.
double	normalizedTF(WordBag bag, String term) Compute TF (term frequency) metrics.
HashMap<String,Double>	normalizeToUnitVector(HashMap<String,Double> vector) Normalize the vector to the norm of 1.
HashMap<String,HashMap<String,Double>>	normalizeToUnitVectors(HashMap<String,HashMap<String,Double>> data) Computes a cosine normalization of TFIDF statistics.
WordBag	seriesToWordBag(String label, double[] ts, Params params) Converts time series to a word bag.
String	tfidfToTable(HashMap<String,HashMap<String,Double>> tfidf)
String	wordBagToTable(WordBag bag)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextProcessor

public TextProcessor()

Constructor.

Method Detail

seriesToWordBag

public WordBag seriesToWordBag(String label,
                               double[] ts,
                               Params params)
                        throws net.seninp.jmotif.sax.SAXException

Converts time series to a word bag.

Parameters:

label - the wordbag label.

ts - timeseries.

params - parameters for SAX transform.

Returns:

word bag.

Throws:

net.seninp.jmotif.sax.SAXException - if error occurs.

labeledSeries2WordBags

public List<WordBag> labeledSeries2WordBags(Map<String,List<double[]>> data,
                                            Params params)
                                     throws net.seninp.jmotif.sax.SAXException

Throws:

net.seninp.jmotif.sax.SAXException

computeTFIDF

public HashMap<String,HashMap<String,Double>> computeTFIDF(Collection<WordBag> texts)

Computes TF*IDF values.

Parameters:

texts - The collection of text documents for which the statistics need to be computed.

Returns:

The map of source documents names to the word - tf*idf weight collections.

logTF

public double logTF(WordBag bag,
                    String term)

Compute TF (term frequency) metrics. This is logarithmically scaled TF.

Parameters:

bag - The words bag.

term - The term.

Returns:

The term frequency value.

normalizedTF

public double normalizedTF(WordBag bag,
                           String term)

Compute TF (term frequency) metrics. This is normalized TF without bias towards longer documents.

Parameters:

bag - The words bag.

term - The term.

Returns:

The term frequency value.

augmentedTF

public double augmentedTF(WordBag bag,
                          String term)

Compute TF (term frequency) metrics. This is normalized TF without bias towards longer documents.

Parameters:

bag - The words bag.

term - The term.

Returns:

The term frequency value.

logAveTF

public double logAveTF(WordBag bag,
                       String term)

Compute TF (term frequency) metrics. This is normalized TF without bias towards longer documents.

Parameters:

bag - The words bag.

term - The term.

Returns:

The term frequency value.

df

public int df(HashMap<String,WordBag> bags,
              String string)

Compute document frequency, DF, metrics.

Parameters:

bags - The word bags collection.

string - The string term.

Returns:

The DF value.

idf

public double idf(HashMap<String,WordBag> bags,
                  String string)

Compute idf (inverse document frequency) metrics.

Parameters:

bags - The bags of words collection.

string - The string (term).

Returns:

The idf value.

tfidfToTable

public String tfidfToTable(HashMap<String,HashMap<String,Double>> tfidf)

normalizeToUnitVector

public HashMap<String,Double> normalizeToUnitVector(HashMap<String,Double> vector)

Normalize the vector to the norm of 1.

Parameters:

vector - the vector.

Returns:

normalized vector.

normalizeToUnitVectors

public HashMap<String,HashMap<String,Double>> normalizeToUnitVectors(HashMap<String,HashMap<String,Double>> data)

Computes a cosine normalization of TFIDF statistics.

Parameters:

data - The data.

Returns:

The normalized tfidf statistics.

cosineDistance

public double cosineDistance(HashMap<String,Double> map1,
                             HashMap<String,Double> map2)

Computes a cosine similarity.

Parameters:

map1 - The data vector 1.

map2 - The data vector 2.

Returns:

The cosine distance.

cosineSimilarity

public double cosineSimilarity(WordBag testSample,
                               HashMap<String,Double> weightVector)

getCosineDistanceMatrix

public CosineDistanceMatrix getCosineDistanceMatrix(HashMap<String,HashMap<String,Double>> tfidf)

magnitude

public double magnitude(double[] vector)

Compute the magnitude of the vector.

Parameters:

vector - The vector.

Returns:

The magnitude.

magnitude

public double magnitude(Double[] vector)

Compute the magnitude of the vector.

Parameters:

vector - The vector.

Returns:

The magnitude.

cosineSimilarityInstrumented

public double cosineSimilarityInstrumented(WordBag testSample,
                                           HashMap<String,Double> weightVector,
                                           HashMap<String,Double> insight)

dotProduct

public double dotProduct(double[] vector1,
                         double[] vector2)

Compute the dot product of two vectors.

Parameters:

vector1 - The vector 1.

vector2 - The vector 2.

Returns:

The dot product.

dotProduct

public double dotProduct(Double[] vector1,
                         Double[] vector2)

Compute the dot product of two vectors.

Parameters:

vector1 - The vector 1.

vector2 - The vector 2.

Returns:

The dot product.

classify

public int classify(String classKey,
                    double[] series,
                    HashMap<String,HashMap<String,Double>> tfidf,
                    Params params)
             throws net.seninp.jmotif.sax.SAXException

Throws:

net.seninp.jmotif.sax.SAXException

classify

public int classify(String trueClassLabel,
                    WordBag test,
                    HashMap<String,HashMap<String,Double>> tfidf)

classify

public String classify(WordBag test,
                       HashMap<String,HashMap<String,Double>> tfidf)

wordBagToTable

public String wordBagToTable(WordBag bag)

bagsToTable

public String bagsToTable(List<WordBag> bags)