E - The type of objects being classified by the evaluated classifier.public class RankedClassifierEvaluator<E> extends BaseClassifierEvaluator<E>
RankedClassifierEvaluator provides an evaluation harness for
ranked classifiers. It extends the base classifier evaluator with
ranking-specific evaluation metrics.
| Constructor and Description |
|---|
RankedClassifierEvaluator(RankedClassifier<E> classifier,
String[] categories,
boolean storeInputs)
Construct a ranked classifier evaluator with the specified
classifier, categories and flag indicating whether or not to store
inputs.
|
| Modifier and Type | Method and Description |
|---|---|
double |
averageRank(String refCategory,
String responseCategory)
Returns the average rank of the specified response category for
test cases with the specified reference category.
|
double |
averageRankReference()
Returns the average over all test samples of the rank of
the the response that matches the reference category.
|
RankedClassifier<E> |
classifier()
Return the classifier being evaluated.
|
void |
handle(Classified<E> classified)
Add the specified classified object to this evaluator.
|
double |
meanReciprocalRank()
Returns the average over all test samples of the reciprocal of
one plus the rank of the reference category in the response.
|
int |
rankCount(String referenceCategory,
int rank)
Returns the number of times that the reference category's
rank was the specified rank.
|
void |
setClassifier(RankedClassifier<E> classifier)
Set the classifier being evaluated to the specified value.
|
addClassification, categories, confusionMatrix, falseNegatives, falsePositives, numCases, numCategories, oneVersusAll, setClassifier, toString, trueNegatives, truePositivespublic RankedClassifierEvaluator(RankedClassifier<E> classifier, String[] categories, boolean storeInputs)
classifier - Classifier to evaluate.categories - Complete list of categories.storeInputs - Set to true to store input objects.public void setClassifier(RankedClassifier<E> classifier)
classifier - New classifier for this evaluation.IllegalArgumentException - If called from an evaluator with
a runtime type other than RankedClassifierEvaluator.public RankedClassifier<E> classifier()
classifier in class BaseClassifierEvaluator<E>public void handle(Classified<E> classified)
BaseClassifierEvaluatorhandle in interface ObjectHandler<Classified<E>>handle in class BaseClassifierEvaluator<E>classified - Classified object to add to evaluation.public int rankCount(String referenceCategory, int rank)
For example, in the set of training samples and results
described in the method documentation for averageRank(String,String), sample rank counts are as
follows:
rankCount("a",0) = 3
rankCount("a",1) = 1
rankCount("a",2) = 0
rankCount("b",0) = 1
rankCount("b",1) = 0
rankCount("b",2) = 1
rankCount("c",0) = 1
rankCount("c",1) = 0
rankCount("c",2) = 0
referenceCategory - Reference category.rank - Rank of count.IllegalArgumentException - If the category is unknown.public double averageRankReference()
Using the example classifications shown in the method
documentation of averageRank(String,String):
averageRankReference()
= (0 + 0 + 0 + 1 + 0 + 2 + 0)/7 ~ 0.43
public double meanReciprocalRank()
Using the example classifications shown in the method
documentation of averageRank(String,String):
averageRankReference()
= (1/1 + 1/1 + 1/1 + 1/2 + 1/1 + 1/3 + 1/1)/7 ~ 0.83
public double averageRank(String refCategory, String responseCategory)
Double.NaN.
Better classifiers return lower values when the reference and response categories are the same and higher values when they are different.
For example, suppose there are three categories,
a, b and c. Consider
the following seven test cases, with the specified ranked
results:
for which:
Test Case Reference Rank 0 Rank 1 Rank 2 0 a a b c 1 a a c b 2 a a b c 3 a b a c 4 b b a c 5 b a c b 6 c c b a
averageRank("a","a") = (0 + 0 + 0 + 1)/4 = 0.25
averageRank("a","b") = (1 + 2 + 1 + 0)/4 = 1.00
averageRank("a","c") = (2 + 1 + 2 + 2)/4 = 1.75
averageRank("b","a") = (1 + 0)/2 = 0.50
averageRank("b","b") = (0 + 2)/2 = 1.0
averageRank("b","c") = (2 + 1)/2 = 1.5
averageRank("c","a") = (2)/1 = 2.0
averageRank("c","b") = (1)/1 = 1.0
averageRank("c","c") = (0)/1 = 0.0
If every ranked result is complete in assigning every category to a rank, the sum of the average ranks will be one less than the number of cases with the specified reference value. If categories are missing from ranked results, the sums may possible be larger than one minus the number of test cases.
Note that the confusion matrix is computed using only the reference and first column of this matrix of results.
refCategory - Reference category.responseCategory - Response category.IllegalArgumentException - If either category is unknown.Copyright © 2019 Alias-i, Inc.. All rights reserved.