de.datexis.sector

Class SectorAnnotator

java.lang.Object

de.datexis.annotator.Annotator

de.datexis.sector.SectorAnnotator

public class SectorAnnotator
extends Annotator

An Annotator that detects sections in a Document and assigns labels. Implementation of: Sebastian Arnold, Rudolf Schneider, Philippe Cudré-Mauroux, Felix A. Gers and Alexander Löser: "SECTOR: A Neural Model for Coherent Topic Segmentation and Classification." Transactions of the Association for Computational Linguistics (2019).

Author:

Sebastian Arnold

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SectorAnnotator.Builder Builder pattern for creating new SECTOR Annotators.
static class	SectorAnnotator.SegmentationMethod

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	log

Fields inherited from class de.datexis.annotator.Annotator

components, it, provenance, tagger

Constructor Summary

Constructors

Modifier	Constructor and Description
	SectorAnnotator() used for JSON deserialization
protected	SectorAnnotator(AnnotatorComponent comp)
	SectorAnnotator(Tagger root)

Method Summary

Modifier and Type	Method and Description
void	annotate(Collection<Document> docs) Annotate given Documents using SECTOR, i.e. attach SectorAnnotator vectors to sentences.
void	annotate(Collection<Document> docs, SectorAnnotator.SegmentationMethod segmentation) Annotate given Documents using SECTOR, i.e. attach SectorEncoder vectors to sentences.
protected static void	attachVectorsToAnnotations(Document doc, LookupCacheEncoder targetEncoder) Add vectors and class labels for all existing GOLD and PRED annotations.
protected static org.nd4j.linalg.api.ndarray.INDArray	deltaMatrix(org.nd4j.linalg.api.ndarray.INDArray data) Returns a matrix [Tx1] that contains cosine distances between time steps.
protected static org.nd4j.linalg.api.ndarray.INDArray	detectEdges(org.nd4j.linalg.api.ndarray.INDArray dev) Returns a matrix [Tx1] that contains edges in deviation.
protected static org.nd4j.linalg.api.ndarray.INDArray	detectEdges(org.nd4j.linalg.api.ndarray.INDArray dev, int count) Returns a matrix [Tx1] that contains edges with given count in deviation.
protected void	detectSections(Collection<Document> docs, SectorAnnotator.SegmentationMethod segmentation)
protected static org.nd4j.linalg.api.ndarray.INDArray	deviation(org.nd4j.linalg.api.ndarray.INDArray target) Returns a matrix [Tx1] that contains cosine distances between t-1 and t.
protected static org.nd4j.linalg.api.ndarray.INDArray	deviation(org.nd4j.linalg.api.ndarray.INDArray fw, org.nd4j.linalg.api.ndarray.INDArray bw) Returns a matrix [Tx1] that contains cosine distances between forward and backward layer.
double	evaluateModel(Dataset test) Evaluate SECTOR model using a given Dataset.
double	evaluateModel(Dataset test, boolean evalSentenceClassification, boolean evalSegmentation, boolean evalSegmentClassification) Evaluate SECTOR model using a given Dataset.
protected static org.nd4j.linalg.api.ndarray.INDArray	gaussianSmooth(org.nd4j.linalg.api.ndarray.INDArray target)
protected static org.nd4j.linalg.api.ndarray.INDArray	gaussianSmooth(org.nd4j.linalg.api.ndarray.INDArray target, double sd)
protected static org.nd4j.linalg.api.ndarray.INDArray	getEmbeddingMatrix(Document doc)
protected static org.nd4j.linalg.api.ndarray.INDArray	getLayerMatrix(Document doc, Class layerClass)
protected static org.nd4j.linalg.api.ndarray.INDArray	getLayerMatrix(Document doc, String layerClass)
SectorTagger	getTagger()
LookupCacheEncoder	getTargetEncoder()
protected static org.nd4j.linalg.api.ndarray.INDArray	pca(org.nd4j.linalg.api.ndarray.INDArray m, int dimensions)
void	segment(Collection<Document> docs, SectorAnnotator.SegmentationMethod segmentation, boolean mergeSections) Attach SectionAnnotations to each Document with a given segmentation strategy.
void	trainModel(Dataset train) Train a SECTOR model with configured number of epochs.
void	trainModel(Dataset train, int numEpochs) Train a SECTOR model with given fixed number of epochs.
void	trainModelEarlyStopping(Dataset train, Dataset validation, int minEpochs, int minEpochsNoImprovement, int maxEpochs) Train a SECTOR model with early stopping based on MAP score.

Methods inherited from class de.datexis.annotator.Annotator

addComponent, annotate, annotate, annotate, createDataset, createDocument, getComponent, getProvenance, isModelAvailable, isModelAvailableInChildren, readModel, writeComponents, writeHTML, writeModel, writeModel, writeTestLog, writeTrainLog

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

Constructor Detail

SectorAnnotator

public SectorAnnotator()

used for JSON deserialization

SectorAnnotator

public SectorAnnotator(Tagger root)

SectorAnnotator

protected SectorAnnotator(AnnotatorComponent comp)

Method Detail

getTagger

public SectorTagger getTagger()

Overrides:

getTagger in class Annotator

getTargetEncoder

public LookupCacheEncoder getTargetEncoder()

annotate

public void annotate(Collection<Document> docs)

Annotate given Documents using SECTOR, i.e. attach SectorAnnotator vectors to sentences. This function also attaches SectionAnnotations to each Document by using the BEMD strategy.

Overrides:

annotate in class Annotator

annotate

public void annotate(Collection<Document> docs,
                     SectorAnnotator.SegmentationMethod segmentation)

Annotate given Documents using SECTOR, i.e. attach SectorEncoder vectors to sentences. If a segmentation method is given, also attach SectionAnnotations to each Document.

segment

public void segment(Collection<Document> docs,
                    SectorAnnotator.SegmentationMethod segmentation,
                    boolean mergeSections)

Attach SectionAnnotations to each Document with a given segmentation strategy. If there are no SectorEncoder vectors attached to sentences yet, please use annotate().

detectSections

protected void detectSections(Collection<Document> docs,
                              SectorAnnotator.SegmentationMethod segmentation)

evaluateModel

public double evaluateModel(Dataset test)

Evaluate SECTOR model using a given Dataset. This method will print a result table.

Returns:

MAP score for segment-level evaluation.

evaluateModel

public double evaluateModel(Dataset test,
                            boolean evalSentenceClassification,
                            boolean evalSegmentation,
                            boolean evalSegmentClassification)

Evaluate SECTOR model using a given Dataset. This method will print a result table.

Parameters:

evalSentenceClassification - - enable/disable the evaluation of sentence-level classification (P/R scores)

evalSegmentation - - enable/disable the evaluation of text segmentation (Pk/WD scores)

evalSegmentClassification - - enable/disable the evaluation of segment-level classification (P/R scores)

Returns:

MAP score for segment-level evaluation.

trainModel

public void trainModel(Dataset train)

Train a SECTOR model with configured number of epochs.

Overrides:

trainModel in class Annotator

trainModel

public void trainModel(Dataset train,
                       int numEpochs)

Train a SECTOR model with given fixed number of epochs.

trainModelEarlyStopping

public void trainModelEarlyStopping(Dataset train,
                                    Dataset validation,
                                    int minEpochs,
                                    int minEpochsNoImprovement,
                                    int maxEpochs)

Train a SECTOR model with early stopping based on MAP score. The best model will be used after this call.

Parameters:

train - training Dataset with GOLD Annotations

validation - validation Dataset with GOLD Annotations

minEpochs - training will not be stopped before this number of epochs (absolute value)

minEpochsNoImprovement - training will be stopped after this number of epochs without a MAP improvement (relative value)

maxEpochs - training will be stopped after this number of epochs (absolute value)

attachVectorsToAnnotations

protected static void attachVectorsToAnnotations(Document doc,
                                                 LookupCacheEncoder targetEncoder)

Add vectors and class labels for all existing GOLD and PRED annotations.

getLayerMatrix

protected static org.nd4j.linalg.api.ndarray.INDArray getLayerMatrix(Document doc,
                                                                     String layerClass)

Returns:

Matrix sentences x layersize that contains target predictions for a Document

getLayerMatrix

protected static org.nd4j.linalg.api.ndarray.INDArray getLayerMatrix(Document doc,
                                                                     Class layerClass)

getEmbeddingMatrix

protected static org.nd4j.linalg.api.ndarray.INDArray getEmbeddingMatrix(Document doc)

Returns:

Matrix sentences x layersize that contains Sector embeddings for a Document

pca

protected static org.nd4j.linalg.api.ndarray.INDArray pca(org.nd4j.linalg.api.ndarray.INDArray m,
                                                          int dimensions)

gaussianSmooth

protected static org.nd4j.linalg.api.ndarray.INDArray gaussianSmooth(org.nd4j.linalg.api.ndarray.INDArray target)

gaussianSmooth

protected static org.nd4j.linalg.api.ndarray.INDArray gaussianSmooth(org.nd4j.linalg.api.ndarray.INDArray target,
                                                                     double sd)

deviation

protected static org.nd4j.linalg.api.ndarray.INDArray deviation(org.nd4j.linalg.api.ndarray.INDArray fw,
                                                                org.nd4j.linalg.api.ndarray.INDArray bw)

Returns a matrix [Tx1] that contains cosine distances between forward and backward layer.

deviation

protected static org.nd4j.linalg.api.ndarray.INDArray deviation(org.nd4j.linalg.api.ndarray.INDArray target)

Returns a matrix [Tx1] that contains cosine distances between t-1 and t.

detectEdges

protected static org.nd4j.linalg.api.ndarray.INDArray detectEdges(org.nd4j.linalg.api.ndarray.INDArray dev)

Returns a matrix [Tx1] that contains edges in deviation.

detectEdges

protected static org.nd4j.linalg.api.ndarray.INDArray detectEdges(org.nd4j.linalg.api.ndarray.INDArray dev,
                                                                  int count)

Returns a matrix [Tx1] that contains edges with given count in deviation.

deltaMatrix

protected static org.nd4j.linalg.api.ndarray.INDArray deltaMatrix(org.nd4j.linalg.api.ndarray.INDArray data)

Returns a matrix [Tx1] that contains cosine distances between time steps.