de.l3s.icrawl.contentanalysis

Class LanguageModels

java.lang.Object

de.l3s.icrawl.contentanalysis.LanguageModels

public class LanguageModels
extends Object

Constructor Summary

Constructors

Constructor and Description
LanguageModels(Locale language, Map<Locale,LanguageModel> modelsMap)
LanguageModels(Locale language, Map<String,Double> idfDictionary, Locale defaultLanguage)

Method Summary

Modifier and Type	Method and Description
DocumentVector	buildDocumentVector(Locale language, String document, de.l3s.icrawl.contentanalysis.LanguageModel.KeywordMatcher keywordMatcher)
de.l3s.icrawl.contentanalysis.LanguageModel.KeywordMatcher	buildMatcher(Locale lang, Iterable<String> keywords, int ngramSize)
Locale	getDefaultLanguage()
double	getSimilarity(Locale language, String doc, DocumentVector reference, de.l3s.icrawl.contentanalysis.LanguageModel.KeywordMatcher matcher) calculate the cosine-similarity of the doc to the specification
static Map<String,Double>	readIdfDictionary(InputStream is)
static LanguageModel	readLanguageModel(Locale locale, InputStream idfIs)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LanguageModels

public LanguageModels(Locale language,
                      Map<String,Double> idfDictionary,
                      Locale defaultLanguage)

LanguageModels

public LanguageModels(Locale language,
                      Map<Locale,LanguageModel> modelsMap)

Method Detail

getDefaultLanguage

public Locale getDefaultLanguage()

getSimilarity

public double getSimilarity(Locale language,
                            String doc,
                            DocumentVector reference,
                            de.l3s.icrawl.contentanalysis.LanguageModel.KeywordMatcher matcher)

calculate the cosine-similarity of the doc to the specification

Parameters:

language - the language of this document

doc - the text of the document

Returns:

the cosine-similarity of the doc to the specification

buildMatcher

public de.l3s.icrawl.contentanalysis.LanguageModel.KeywordMatcher buildMatcher(Locale lang,
                                                                               Iterable<String> keywords,
                                                                               int ngramSize)

buildDocumentVector

public DocumentVector buildDocumentVector(Locale language,
                                          String document,
                                          de.l3s.icrawl.contentanalysis.LanguageModel.KeywordMatcher keywordMatcher)

readIdfDictionary

public static Map<String,Double> readIdfDictionary(InputStream is)
                                            throws IOException

Throws:

IOException

readLanguageModel

public static LanguageModel readLanguageModel(Locale locale,
                                              InputStream idfIs)
                                       throws IOException

Throws:

IOException