de.datexis.ner.eval

Class MentionAnnotatorEvaluation

java.lang.Object

de.datexis.annotator.AnnotatorEvaluation

de.datexis.ner.eval.MentionAnnotatorEvaluation

public class MentionAnnotatorEvaluation
extends AnnotatorEvaluation

Evaluates Precision/Recall/F1 for Annotation Matching.

Author:

Sebastian Arnold

Nested Class Summary

Nested classes/interfaces inherited from class de.datexis.annotator.AnnotatorEvaluation

AnnotatorEvaluation.Measure

Field Summary

Fields

Modifier and Type	Field and Description
protected TreeMap<AnnotatorEvaluation.Measure,org.nd4j.linalg.primitives.Counter<Integer>>	counts
protected static org.slf4j.Logger	log

Fields inherited from class de.datexis.annotator.AnnotatorEvaluation

countAnnotations, countDocs, countExamples, countSentences, countTokens, expectedSource, experimentName, predictedSource

Constructor Summary

Constructors

Constructor and Description
MentionAnnotatorEvaluation(String experimentName, Annotation.Match matchingStrategy)
MentionAnnotatorEvaluation(String experimentName, Annotation.Source expected, Annotation.Source predicted, Annotation.Match matchingStrategy)

Method Summary

Modifier and Type	Method and Description
void	calculateScores(Collection<Document> docs)
void	calculateScores(Dataset dataset)
void	calculateScoresFromAnnotations(Collection<Document> docs, Class<? extends Annotation> annotationClass)
protected void	clear()
protected double	div(double n, double d) safe division, where n/0 = 0
protected void	fixCounters() required to update totalCount() - fixed in next Nd4j https://github.com/deeplearning4j/nd4j/commit/2698b2e23d8ccf6cf71c3bf6fc325e9638877ae8
double	getAccuracy() Micro/Macro Accuracy
protected double	getAccuracy(int c) Accuracy per class
protected double	getCount(AnnotatorEvaluation.Measure m, int classIdx)
protected double	getF1(int i) F1 score per document
double	getFN()
double	getFN(Document d, Class<? extends Annotation> annotationClass)
double	getFP()
double	getFP(Document d, Class<? extends Annotation> annotationClass)
double	getMacroF1() Macro F1 score (average F1 over all classes).
double	getMacroPrecision() Macro Precision (average Precision over all classes).
double	getMacroRecall() Macro Recall (average recall over all classes).
double	getMicroF1() Micro F1 score (average F1 over all examples).
double	getMicroPrecision() Micro Precision (average precision over all examples).
double	getMicroRecall() Micro Recall (average recall over all examples).
protected double	getPrecision(int c) Precision per class
protected double	getRecall(int c) Recall per class
double	getScore()
double	getTN()
double	getTN(Document d, Class<? extends Annotation> annotationClass)
double	getTokenAccuracy(Dataset data) This is CoNLL2003 Accuracy
double	getTP()
double	getTP(Document d, Class<? extends Annotation> annotationClass)
String	printAnnotationStats()
static String	printHeader()
String	printRow()

Methods inherited from class de.datexis.annotator.AnnotatorEvaluation

countAnnotations, countDocuments, countExamples, countSentences, countTokens, fDbl, fDbl4, fInt, fStr, printEvaluationStats

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static org.slf4j.Logger log

counts

protected TreeMap<AnnotatorEvaluation.Measure,org.nd4j.linalg.primitives.Counter<Integer>> counts

Constructor Detail

MentionAnnotatorEvaluation

public MentionAnnotatorEvaluation(String experimentName,
                                  Annotation.Match matchingStrategy)

MentionAnnotatorEvaluation

public MentionAnnotatorEvaluation(String experimentName,
                                  Annotation.Source expected,
                                  Annotation.Source predicted,
                                  Annotation.Match matchingStrategy)

Method Detail

clear

protected void clear()

getCount

protected double getCount(AnnotatorEvaluation.Measure m,
                          int classIdx)

getScore

public double getScore()

Specified by:

getScore in class AnnotatorEvaluation

calculateScores

public void calculateScores(Dataset dataset)

Overrides:

calculateScores in class AnnotatorEvaluation

calculateScores

public void calculateScores(Collection<Document> docs)

Specified by:

calculateScores in class AnnotatorEvaluation

calculateScoresFromAnnotations

public void calculateScoresFromAnnotations(Collection<Document> docs,
                                           Class<? extends Annotation> annotationClass)

fixCounters

protected void fixCounters()

required to update totalCount() - fixed in next Nd4j https://github.com/deeplearning4j/nd4j/commit/2698b2e23d8ccf6cf71c3bf6fc325e9638877ae8

getTP

public double getTP(Document d,
                    Class<? extends Annotation> annotationClass)

getFP

public double getFP(Document d,
                    Class<? extends Annotation> annotationClass)

getTN

public double getTN(Document d,
                    Class<? extends Annotation> annotationClass)

getFN

public double getFN(Document d,
                    Class<? extends Annotation> annotationClass)

getTP

public double getTP()

getFP

public double getFP()

getTN

public double getTN()

getFN

public double getFN()

div

protected double div(double n,
                     double d)

safe division, where n/0 = 0

Overrides:

div in class AnnotatorEvaluation

getTokenAccuracy

public double getTokenAccuracy(Dataset data)

This is CoNLL2003 Accuracy

Parameters:

data -

Returns:

accuracy = correctTags / tokenCounter

getAccuracy

public double getAccuracy()

Micro/Macro Accuracy

getAccuracy

protected double getAccuracy(int c)

Accuracy per class

Parameters:

c - - class index

getMicroPrecision

public double getMicroPrecision()

Micro Precision (average precision over all examples). This is the CoNLL2003 Precision.

Returns:

precision = correctChunk / foundGuessed

getMacroPrecision

public double getMacroPrecision()

Macro Precision (average Precision over all classes).

getPrecision

protected double getPrecision(int c)

Precision per class

Parameters:

c - - class index

getMicroRecall

public double getMicroRecall()

Micro Recall (average recall over all examples). This is the CoNLL2003 Recall.

Returns:

recall = correctChunk / foundCorrect

getMacroRecall

public double getMacroRecall()

Macro Recall (average recall over all classes).

getRecall

protected double getRecall(int c)

Recall per class

Parameters:

c - - class index

getMicroF1

public double getMicroF1()

Micro F1 score (average F1 over all examples). This is CoNLL2003 NER-style F1

Returns:

$FB1 = 2*$precision*$recall/($precision+$recall) if ($precision+$recall > 0)

getMacroF1

public double getMacroF1()

Macro F1 score (average F1 over all classes).

getF1

protected double getF1(int i)

F1 score per document

Parameters:

i - - document index

printAnnotationStats

public String printAnnotationStats()

printHeader

public static String printHeader()

printRow

public String printRow()