de.datexis.sector.eval

Class SectorEvaluation

java.lang.Object

de.datexis.annotator.AnnotatorEvaluation

de.datexis.sector.eval.SectorEvaluation

public class SectorEvaluation
extends AnnotatorEvaluation

Combined Evaluation of Headings per Sentence, Segmentation and Classification per Segment

Author:

Sebastian Arnold

Nested Class Summary

Nested classes/interfaces inherited from class de.datexis.annotator.AnnotatorEvaluation

AnnotatorEvaluation.Measure

Field Summary

Fields

Modifier and Type	Field and Description
protected int	countPredictions
protected int	countSections
protected boolean	enableSegmentationEval
protected boolean	enableSegmentEval
protected boolean	enableSentenceEval
protected static org.slf4j.Logger	log
protected SegmentationEvaluation	segmentationEval
protected ClassificationEvaluation	segmentClassEval
protected ClassificationEvaluation	sentenceClassEval
protected Class	targetEncoderClass

Fields inherited from class de.datexis.annotator.AnnotatorEvaluation

countAnnotations, countDocs, countExamples, countSentences, countTokens, expectedSource, experimentName, predictedSource

Constructor Summary

Constructors

Constructor and Description
SectorEvaluation(String experimentName, Annotation.Source expected, Annotation.Source predicted) Initialize evaluation for segmentation only
SectorEvaluation(String experimentName, Annotation.Source expected, Annotation.Source predicted, LookupCacheEncoder targetEncoder) Initialize evaluation for classification and segmentation

Method Summary

Modifier and Type	Method and Description
void	calculateScores(Collection<Document> docs) Evaluate the whole SECTOR model.
double	getScore() Return the MAP score from this Evaluation run.
static String	printDatasetStats(Dataset dataset) Print out some stats about the Dataset.
String	printEvaluationStats() Print out the Evaluation results table.
String	printSingleClassStats() Print out Evaluation results table for single classes.
static String	printSingleClassStats(ClassificationEvaluation eval)
SectorEvaluation	withMergeEnabled(boolean enabled) Enable/disable merging adjacent sections with same label into one (in both GOLD and PRED).
SectorEvaluation	withRecalculateK(boolean enabled) Enable/disable recalculation of K for each document in the evaluation.
SectorEvaluation	withSegmentationEvaluation(boolean enable) Enable/disable evaluation for segmentation (Pk/WD).
SectorEvaluation	withSegmentClassEvaluation(boolean enable) Enable/disable evaluation for segment-based classification (Prec/Rec/F1).
SectorEvaluation	withSentenceClassEvaluation(boolean enable) Enable/disable evaluation for sentence-based classification (Prec/Rec/F1).

Methods inherited from class de.datexis.annotator.AnnotatorEvaluation

calculateScores, countAnnotations, countDocuments, countExamples, countSentences, countTokens, div, fDbl, fInt, fStr

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

enableSentenceEval

protected boolean enableSentenceEval

enableSegmentEval

protected boolean enableSegmentEval

enableSegmentationEval

protected boolean enableSegmentationEval

countSections

protected int countSections

countPredictions

protected int countPredictions

sentenceClassEval

protected ClassificationEvaluation sentenceClassEval

segmentClassEval

protected ClassificationEvaluation segmentClassEval

segmentationEval

protected SegmentationEvaluation segmentationEval

targetEncoderClass

protected Class targetEncoderClass

Constructor Detail

SectorEvaluation

public SectorEvaluation(String experimentName,
                        Annotation.Source expected,
                        Annotation.Source predicted,
                        LookupCacheEncoder targetEncoder)

Initialize evaluation for classification and segmentation

SectorEvaluation

public SectorEvaluation(String experimentName,
                        Annotation.Source expected,
                        Annotation.Source predicted)

Initialize evaluation for segmentation only

Method Detail

withSentenceClassEvaluation

public SectorEvaluation withSentenceClassEvaluation(boolean enable)

Enable/disable evaluation for sentence-based classification (Prec/Rec/F1).

withSegmentationEvaluation

public SectorEvaluation withSegmentationEvaluation(boolean enable)

Enable/disable evaluation for segmentation (Pk/WD).

withSegmentClassEvaluation

public SectorEvaluation withSegmentClassEvaluation(boolean enable)

Enable/disable evaluation for segment-based classification (Prec/Rec/F1).

withRecalculateK

public SectorEvaluation withRecalculateK(boolean enabled)

Enable/disable recalculation of K for each document in the evaluation. if FALSE, we use a fixed K for all documents.

withMergeEnabled

public SectorEvaluation withMergeEnabled(boolean enabled)

Enable/disable merging adjacent sections with same label into one (in both GOLD and PRED).

calculateScores

public void calculateScores(Collection<Document> docs)

Evaluate the whole SECTOR model. For Multi-label evaluation of Headings: - requires expected and predicted HeadingTags attached to every Sentence in the test set - requires class distribution Vectors attached to the HeadingTags For Segmentation evaluation: - requires expected and predicted SectionAnnotations attached to Documents For Classification evaluation: - requires class distribution Vectors attached to expected and predicted Annotations

Specified by:

calculateScores in class AnnotatorEvaluation

getScore

public double getScore()

Return the MAP score from this Evaluation run.

Specified by:

getScore in class AnnotatorEvaluation

printDatasetStats

public static String printDatasetStats(Dataset dataset)

Print out some stats about the Dataset.

printEvaluationStats

public String printEvaluationStats()

Print out the Evaluation results table.

Overrides:

printEvaluationStats in class AnnotatorEvaluation

printSingleClassStats

public String printSingleClassStats()

Print out Evaluation results table for single classes.

printSingleClassStats

public static String printSingleClassStats(ClassificationEvaluation eval)