com.aliasi.chunk

Class ChunkingEvaluation

java.lang.Object

com.aliasi.chunk.ChunkingEvaluation

public class ChunkingEvaluation
extends Object

A ChunkingEvaluation stores and reports the results of evaluating response chunkings against reference chunkings. Cases to evaluate are supplied in the form of a reference and response chunking through the method addCase(Chunking,Chunking).

The sets of true positive, false positive and false negative chunks are available through the methods truePositiveSet(), falsePositiveSet(), and falseNegativeSet(). True positives are chunks that are in both the reference and response, false positives are chunks in the response but not the reference, and false negatives are in the reference, but not the response. There is no notion of true negative in this task, a fact that is reflected in the results of the precision-recall evaluation.

The main method of reporting is through an instance of ScoredPrecisionRecallEvaluation returned by the method precisionRecallEvaluation(). The return result provides an object capable of extensive reporting for scored classification tasks such as chunking. The instances of true and false positive and negatives are described above; their scores are derived from response scores.

This evaluator works solely on the basis of chunk offset and exact match. There is no notion of alignment or mapping, as found, for example, in the MUC Scoring Software User's Manual, and its descendants such as the 2005 ACE Evaluation Plan. In this regard, we follow the model of CoNLL 2000 Chunking Task.

This evaluation is able to handle overlapping chunks with results being reported in the same manner. In particular, the labeled precision and recall components of the approach that later became known as PARSEVAL can be generated by using the ChunkingEvaluation class.

Since:

LingPipe2.1

Version:

3.8

Author:

Bob Carpenter

Constructor Summary

Constructors

Constructor and Description
ChunkingEvaluation() Construct a chunking evaluation.

Method Summary

Methods

Modifier and Type	Method and Description
void	addCase(Chunking referenceChunking, Chunking responseChunking) Add an evaluation case consisting of a reference chunk set and a response chunk set.
Set<Chunking[]>	cases() Return the set of cases consisting of pairs of reference and response chunkings.
Set<ChunkAndCharSeq>	falseNegativeSet() Returns the set of false negatives.
Set<ChunkAndCharSeq>	falsePositiveSet() Returns the set of false positives.
ChunkingEvaluation	perTypeEvaluation(String chunkType) Returns a chunking evaluation which consists of the current chunking evaluation restricted to the specified type.
PrecisionRecallEvaluation	precisionRecallEvaluation() Return the scored precision-recall evaluation for this chunker.
String	toString() Returns the precision-recall evaluation for this chunking as a string.
Set<ChunkAndCharSeq>	truePositiveSet() Returns the set of true positives.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ChunkingEvaluation

public ChunkingEvaluation()

Construct a chunking evaluation.

Method Detail

cases

public Set<Chunking[]> cases()

Return the set of cases consisting of pairs of reference and response chunkings. The elements of the set returned are of type Chunking[], with the first element being the reference chunk and the second element being the response chunk.

The set returned is an unmodifiable view of the underlying set of cases and will change as cases are added to this evaluation.

Returns:

The set of cases.

perTypeEvaluation

public ChunkingEvaluation perTypeEvaluation(String chunkType)

Returns a chunking evaluation which consists of the current chunking evaluation restricted to the specified type. A new evaluation is constructed and populated with the same cases as this evaluation, but with the reference and response chunkings both restricted to only include answers of the specified type.

Parameters:

chunkType - Type of chunk to be evaluated.

Returns:

ChunkingEvaluation Evaluation for this type.

addCase

public void addCase(Chunking referenceChunking,
           Chunking responseChunking)

Add an evaluation case consisting of a reference chunk set and a response chunk set.

Parameters:

referenceChunking - Chunking of reference chunks.

responseChunking - Chunking of response chunks.

Throws:

IllegalArgumentException - If the chunkings are not over the same character sequence.

truePositiveSet

public Set<ChunkAndCharSeq> truePositiveSet()

Returns the set of true positives. True positives are chunks that were in both a reference and response chunking case. The set returned contains instances of ChunkAndCharSeq, which combine a chunk and a character sequence.

The set is unmodifiable, but tracks the changes in this evaluator.

Returns:

The set of true positives.

falsePositiveSet

public Set<ChunkAndCharSeq> falsePositiveSet()

Returns the set of false positives. False positives are response chunks that are not reference chunks. The set returned contains instances of ChunkAndCharSeq, which combine a chunk and a character sequence.

The set is unmodifiable, but tracks the changes in this evaluator.

Returns:

The set of false positives.

falseNegativeSet

public Set<ChunkAndCharSeq> falseNegativeSet()

Returns the set of false negatives. False negatives are reference chunks which are not response chunks. The set returned contains instances of ChunkAndCharSeq, which combine a chunk and a character sequence.

The set is unmodifiable, but tracks the changes in this evaluator.

Returns:

The set of false negatives.

precisionRecallEvaluation

public PrecisionRecallEvaluation precisionRecallEvaluation()

Return the scored precision-recall evaluation for this chunker. This is a copy of the precision-recall evaluation and changes to it will not affect the results returned by this class.

Returns:

The precision-recall evaluation.

toString

public String toString()

Returns the precision-recall evaluation for this chunking as a string.

Overrides:

toString in class Object

Returns:

This evaluation as a string.