de.datexis.ner.tagger

Class MentionTagger

java.lang.Object

de.datexis.annotator.AnnotatorComponent

de.datexis.tagger.Tagger

de.datexis.ner.tagger.MentionTagger

All Implemented Interfaces:

IComponent

public class MentionTagger
extends Tagger

Assigns BIO2 or BIOES Labels to every Token in a Document.

Author:

Sebastian Arnold

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	log
protected Class<? extends Tag>	tagset
protected String	type
protected int	workers

Fields inherited from class de.datexis.tagger.Tagger

batchSize, embeddingLayerSize, embeddingVectorSize, inputVectorSize, maxTimeSeriesLength, net, numEpochs, numExamples, outputVectorSize, randomize

Fields inherited from class de.datexis.annotator.AnnotatorComponent

id, model, modelAvailable, name, timer

Constructor Summary

Constructors

Constructor and Description
MentionTagger()
MentionTagger(AbstractIterator data, int ffwLayerSize, int lstmLayerSize, int iterations, double learningRate)
MentionTagger(String id)

Method Summary

Modifier and Type	Method and Description
static org.deeplearning4j.nn.graph.ComputationGraph	createBLSTM(long inputVectorSize, long ffwLayerSize, long lstmLayerSize, long outputVectorSize, int iterations, double learningRate)
static void	createTags(Iterable<Sentence> sents, org.nd4j.linalg.api.ndarray.INDArray predicted, Class tagset, Annotation.Source source, String type, boolean keepVectors, boolean convertTags) Creates BIO2Tags from predictions.
void	enableTrainingUI()
EncoderSet	getEncoderSet() Deprecated.
Class<? extends Tag>	getTagset()
String	getType()
MentionTagger	setModelParams(int ffwLayerSize, int lstmLayerSize, int iterations, double learningRate)
MentionTagger	setTagset(Class<? extends Tag> tagset)
MentionTagger	setTagset(Class<? extends Tag> tagset, String types)
MentionTagger	setTrainingParams(int batchSize, int numEpochs, boolean randomize)
void	setType(String type)
MentionTagger	setWorkspaceParams(int workers)
void	tag(Collection<Document> documents) Predicts labels for all Tokens in the Iterator and assigns Tags (BIO2 or BIOES).
void	tagSentences(Collection<Sentence> sentences)
org.nd4j.evaluation.classification.Evaluation	test(MentionTaggerIterator it)
void	testModel(Dataset dataset, Annotation.Source expected) requires: GOLD BIO2Tag.class and BIOESTag.class for Token.class attaches: PRED BIO2Tag.class and BIOESTag.class to Token.class
void	trainModel(Collection<Sentence> sentences, Annotation.Source trainingTags, boolean randomize)
void	trainModel(Dataset dataset)
void	trainModel(Dataset dataset, Annotation.Source trainingAnnotations)
void	trainModel(Dataset dataset, Annotation.Source trainingAnnotations, int numExamples, boolean randomize)
protected void	trainModel(MentionTaggerIterator it)

Methods inherited from class de.datexis.tagger.Tagger

getBatchSize, getEmbeddingLayerSize, getEncoders, getGraphConfiguration, getLayerConfiguration, getMaxTimeSeriesLength, getNN, getNumEpochs, isModelAvailableInChildren, isRandomize, loadConf, loadModel, saveModel, saveUpdater, setBatchSize, setEmbeddingLayerSize, setEncoders, setGraphConfiguration, setLayerConfiguration, setListeners, setMaxTimeSeriesLength, setNumEpochs, setRandomize, setTrainingParams, tag, testModel

Methods inherited from class de.datexis.annotator.AnnotatorComponent

appendTestLog, appendTestLog, appendTrainLog, appendTrainLog, clearTestLog, clearTrainLog, getConf, getId, getModel, getName, getTestLog, getTrainLog, isModelAvailable, setConf, setId, setModel, setModelAvailable, setModelFilename, setName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

workers

protected int workers

tagset

protected Class<? extends Tag> tagset

type

protected String type

Constructor Detail

MentionTagger

public MentionTagger()

MentionTagger

public MentionTagger(String id)

MentionTagger

public MentionTagger(AbstractIterator data,
                     int ffwLayerSize,
                     int lstmLayerSize,
                     int iterations,
                     double learningRate)

Method Detail

setModelParams

public MentionTagger setModelParams(int ffwLayerSize,
                                    int lstmLayerSize,
                                    int iterations,
                                    double learningRate)

getTagset

public Class<? extends Tag> getTagset()

createBLSTM

public static org.deeplearning4j.nn.graph.ComputationGraph createBLSTM(long inputVectorSize,
                                                                       long ffwLayerSize,
                                                                       long lstmLayerSize,
                                                                       long outputVectorSize,
                                                                       int iterations,
                                                                       double learningRate)

getType

public String getType()

setType

public void setType(String type)

setTagset

public MentionTagger setTagset(Class<? extends Tag> tagset)

setTagset

public MentionTagger setTagset(Class<? extends Tag> tagset,
                               String types)

setTrainingParams

public MentionTagger setTrainingParams(int batchSize,
                                       int numEpochs,
                                       boolean randomize)

setWorkspaceParams

public MentionTagger setWorkspaceParams(int workers)

getEncoderSet

@Deprecated
public EncoderSet getEncoderSet()

Deprecated.

trainModel

public void trainModel(Dataset dataset)

Overrides:

trainModel in class Tagger

trainModel

public void trainModel(Dataset dataset,
                       Annotation.Source trainingAnnotations)

trainModel

public void trainModel(Dataset dataset,
                       Annotation.Source trainingAnnotations,
                       int numExamples,
                       boolean randomize)

trainModel

public void trainModel(Collection<Sentence> sentences,
                       Annotation.Source trainingTags,
                       boolean randomize)

trainModel

protected void trainModel(MentionTaggerIterator it)

tag

public void tag(Collection<Document> documents)

Predicts labels for all Tokens in the Iterator and assigns Tags (BIO2 or BIOES). requires: Encoder.class on Token.class (using parallelized DocumentIterator batches) attaches: BIO2Tag.class to Token.class

Specified by:

tag in class Tagger

tagSentences

public void tagSentences(Collection<Sentence> sentences)

testModel

public void testModel(Dataset dataset,
                      Annotation.Source expected)

requires: GOLD BIO2Tag.class and BIOESTag.class for Token.class attaches: PRED BIO2Tag.class and BIOESTag.class to Token.class

Parameters:

dataset -

expected -

test

public org.nd4j.evaluation.classification.Evaluation test(MentionTaggerIterator it)

enableTrainingUI

public void enableTrainingUI()

createTags

public static void createTags(Iterable<Sentence> sents,
                              org.nd4j.linalg.api.ndarray.INDArray predicted,
                              Class tagset,
                              Annotation.Source source,
                              String type,
                              boolean keepVectors,
                              boolean convertTags)

Creates BIO2Tags from predictions. requires: INDArray predictions for Token.class attaches: BIO2Tag.class to Token.class

Parameters:

sents - List of sentences to tag.

predicted - Predictions from the sequence model.

tagset - The tagset to use for tagging (e.g. BIOES). Output will be transformed to BIOES then.

source - Which tags to use: GOLD, PREDICTED or USER.

keepVectors - If TRUE, vectors are not deleted from the Tokens.

convertTags - If TRUE, tags are corrected and converted to BIO2. Otherwise, we keep Tags of class tagset.