de.datexis.sector.eval

Class ClassificationEvaluation

java.lang.Object

de.datexis.annotator.AnnotatorEvaluation

de.datexis.sector.eval.ClassificationEvaluation

All Implemented Interfaces:

Serializable, org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

public class ClassificationEvaluation
extends AnnotatorEvaluation
implements org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

Evaluates Precision/Recall/F1 for Sentence-based class labeling (e.g. Sentence Classification).

Author:

Sebastian Arnold

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class de.datexis.annotator.AnnotatorEvaluation

AnnotatorEvaluation.Measure

Field Summary

Fields

Modifier and Type	Field and Description
protected LookupCacheEncoder	encoder
protected org.nd4j.evaluation.classification.Evaluation	eval
protected int	K
protected double	mapsum average precision
protected double	mrrsum average precision
protected int	numClasses
protected double	p1sum average precision
protected double	pksum average precision
protected double	r1sum average precision
protected double	rksum average precision

Fields inherited from class de.datexis.annotator.AnnotatorEvaluation

countAnnotations, countDocs, countExamples, countSentences, countTokens, expectedSource, experimentName, log, predictedSource

Constructor Summary

Constructors

Constructor and Description
ClassificationEvaluation(String experimentName, Annotation.Source expected, Annotation.Source predicted, LookupCacheEncoder encoder, int K)
ClassificationEvaluation(String experimentName, LookupCacheEncoder encoder)

Method Summary

Modifier and Type	Method and Description
void	calculateScores(Collection<Document> docs) Not used.
void	calculateScoresFromAnnotations(Collection<Document> documents, Class<? extends Annotation> annotationClass, boolean matchAllPredicted) Calculates Evaluation from Annotations in the Documents
<T extends Tag> void	calculateScoresFromTags(Collection<Document> documents, Class<? extends Span> spanClass, Class<T> tagClass) Calculates Evaluation from Tags in the Spans
protected void	clear()
protected double	div(double n, double d) safe division, where n/0 = 0
void	eval(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray networkPredictions)
void	eval(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray networkPredictions, org.nd4j.linalg.api.ndarray.INDArray maskArray)
void	eval(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray networkPredictions, org.nd4j.linalg.api.ndarray.INDArray maskArray, List<? extends Serializable> recordMetaData)
void	eval(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray networkPredictions, List<? extends Serializable> recordMetaData)
void	evalExample(org.nd4j.linalg.api.ndarray.INDArray Y, org.nd4j.linalg.api.ndarray.INDArray Z) Update scores from a single Example prediction
void	evalTimeSeries(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray predicted)
void	evalTimeSeries(org.nd4j.linalg.api.ndarray.INDArray labels, org.nd4j.linalg.api.ndarray.INDArray predictions, org.nd4j.linalg.api.ndarray.INDArray labelsMask)
double	getAccuracy() Micro/Macro Accuracy
protected double	getAccuracy(int c) Accuracy per class
double	getAccuracyK()
protected double	getF1(int c) F1 score per class
double	getMacroF1() Macro F1 score (average F1 over all classes).
double	getMacroPrecision() Macro Precision (average Precision over all classes).
double	getMacroRecall() Macro Recall (average recall over all classes).
double	getMAP()
double	getMicroF1() Micro F1 score (average F1 over all examples).
double	getMicroPrecision() Micro Precision (average precision over all examples).
double	getMicroRecall() Micro Recall (average recall over all examples).
protected double	getMRR()
protected double	getPrecision(int c) Precision per class
double	getPrecision1()
double	getPrecisionK()
protected double	getRecall(int c) Recall per class
double	getRecall1()
double	getRecallK()
double	getScore()
double	getValue(org.nd4j.evaluation.IMetric iMetric)
protected static int	maxIndex(org.nd4j.linalg.api.ndarray.INDArray Y)
void	merge(ClassificationEvaluation other)
ClassificationEvaluation	newInstance()
String	printClassificationAtKStats() Print micro-averaged scores for evaluation @K
String	printClassificationStats()
protected static int	rank(int idx, org.nd4j.linalg.api.ndarray.INDArray l) get position of index idx in ranked labels l
void	reset()
String	stats()
String	toJson()
String	toYaml()

Methods inherited from class de.datexis.annotator.AnnotatorEvaluation

calculateScores, countAnnotations, countDocuments, countExamples, countSentences, countTokens, fDbl, fDbl4, fInt, fStr, printEvaluationStats

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

encoder

protected LookupCacheEncoder encoder

numClasses

protected int numClasses

K

protected int K

eval

protected org.nd4j.evaluation.classification.Evaluation eval

mrrsum

protected double mrrsum

average precision

mapsum

protected double mapsum

average precision

p1sum

protected double p1sum

average precision

r1sum

protected double r1sum

average precision

pksum

protected double pksum

average precision

rksum

protected double rksum

average precision

Constructor Detail

ClassificationEvaluation

public ClassificationEvaluation(String experimentName,
                                LookupCacheEncoder encoder)

ClassificationEvaluation

public ClassificationEvaluation(String experimentName,
                                Annotation.Source expected,
                                Annotation.Source predicted,
                                LookupCacheEncoder encoder,
                                int K)

Method Detail

clear

protected void clear()

getScore

public double getScore()

Specified by:

getScore in class AnnotatorEvaluation

calculateScores

public void calculateScores(Collection<Document> docs)

Not used. Please use calculateScoresFromAnnotations or calculateScoresFromTags.

Specified by:

calculateScores in class AnnotatorEvaluation

calculateScoresFromAnnotations

public void calculateScoresFromAnnotations(Collection<Document> documents,
                                           Class<? extends Annotation> annotationClass,
                                           boolean matchAllPredicted)

Calculates Evaluation from Annotations in the Documents. - requires expected and predicted Annotations attached to Documents (will match expected to predicted via position) - requires class distribution Vectors attached to expected and predicted Annotations

Parameters:

matchAllPredicted - - if TRUE, all remaining unmatched predicted annotations will be matched to expected via position, otherwise they are ignored

calculateScoresFromTags

public <T extends Tag> void calculateScoresFromTags(Collection<Document> documents,
                                                    Class<? extends Span> spanClass,
                                                    Class<T> tagClass)

Calculates Evaluation from Tags in the Spans. - requires expected and predicted Tags attached to the given Span class - requires class distribution Vectors attached to the Tags

evalExample

public void evalExample(org.nd4j.linalg.api.ndarray.INDArray Y,
                        org.nd4j.linalg.api.ndarray.INDArray Z)

Update scores from a single Example prediction

Parameters:

Y - - correct labels e {0,1}^d

Z - - predicted labels e R^d@param Y

div

protected double div(double n,
                     double d)

safe division, where n/0 = 0

Overrides:

div in class AnnotatorEvaluation

rank

protected static int rank(int idx,
                          org.nd4j.linalg.api.ndarray.INDArray l)

get position of index idx in ranked labels l

Returns:

position between 1 and length(l)

maxIndex

protected static int maxIndex(org.nd4j.linalg.api.ndarray.INDArray Y)

Returns:

the relevant index Yi == 1

getAccuracy

public double getAccuracy()

Micro/Macro Accuracy

getAccuracyK

public double getAccuracyK()

getAccuracy

protected double getAccuracy(int c)

Accuracy per class

Parameters:

c - - class index

getMicroPrecision

public double getMicroPrecision()

Micro Precision (average precision over all examples). This is the CoNLL2003 Precision.

Returns:

precision = correctChunk / foundGuessed

getMacroPrecision

public double getMacroPrecision()

Macro Precision (average Precision over all classes).

getPrecision

protected double getPrecision(int c)

Precision per class

Parameters:

c - - class index

getMicroRecall

public double getMicroRecall()

Micro Recall (average recall over all examples). This is the CoNLL2003 Recall.

Returns:

recall = correctChunk / foundCorrect

getMacroRecall

public double getMacroRecall()

Macro Recall (average recall over all classes).

getRecall

protected double getRecall(int c)

Recall per class

Parameters:

c - - class index

getMicroF1

public double getMicroF1()

Micro F1 score (average F1 over all examples). This is CoNLL2003 NER-style F1

Returns:

$FB1 = 2*$precision*$recall/($precision+$recall) if ($precision+$recall > 0)

getMacroF1

public double getMacroF1()

Macro F1 score (average F1 over all classes).

getF1

protected double getF1(int c)

F1 score per class

Parameters:

c - - class index

getMRR

protected double getMRR()

getMAP

public double getMAP()

getPrecisionK

public double getPrecisionK()

getRecallK

public double getRecallK()

getPrecision1

public double getPrecision1()

getRecall1

public double getRecall1()

eval

public void eval(org.nd4j.linalg.api.ndarray.INDArray labels,
                 org.nd4j.linalg.api.ndarray.INDArray networkPredictions)

Specified by:

eval in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

eval

public void eval(org.nd4j.linalg.api.ndarray.INDArray labels,
                 org.nd4j.linalg.api.ndarray.INDArray networkPredictions,
                 List<? extends Serializable> recordMetaData)

Specified by:

eval in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

eval

public void eval(org.nd4j.linalg.api.ndarray.INDArray labels,
                 org.nd4j.linalg.api.ndarray.INDArray networkPredictions,
                 org.nd4j.linalg.api.ndarray.INDArray maskArray,
                 List<? extends Serializable> recordMetaData)

Specified by:

eval in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

eval

public void eval(org.nd4j.linalg.api.ndarray.INDArray labels,
                 org.nd4j.linalg.api.ndarray.INDArray networkPredictions,
                 org.nd4j.linalg.api.ndarray.INDArray maskArray)

Specified by:

eval in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

evalTimeSeries

public void evalTimeSeries(org.nd4j.linalg.api.ndarray.INDArray labels,
                           org.nd4j.linalg.api.ndarray.INDArray predicted)

Specified by:

evalTimeSeries in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

evalTimeSeries

public void evalTimeSeries(org.nd4j.linalg.api.ndarray.INDArray labels,
                           org.nd4j.linalg.api.ndarray.INDArray predictions,
                           org.nd4j.linalg.api.ndarray.INDArray labelsMask)

Specified by:

evalTimeSeries in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

merge

public void merge(ClassificationEvaluation other)

Specified by:

merge in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

reset

public void reset()

Specified by:

reset in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

stats

public String stats()

Specified by:

stats in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

toJson

public String toJson()

Specified by:

toJson in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

toYaml

public String toYaml()

Specified by:

toYaml in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

getValue

public double getValue(org.nd4j.evaluation.IMetric iMetric)

Specified by:

getValue in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

newInstance

public ClassificationEvaluation newInstance()

Specified by:

newInstance in interface org.nd4j.evaluation.IEvaluation<ClassificationEvaluation>

printClassificationAtKStats

public String printClassificationAtKStats()

Print micro-averaged scores for evaluation @K

Returns:

printClassificationStats

public String printClassificationStats()