de.datexis.ner.tagger

Class MentionTaggerIterator

java.lang.Object

de.datexis.tagger.AbstractIterator

de.datexis.tagger.CachedSentenceIterator

de.datexis.ner.tagger.MentionTaggerIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class MentionTaggerIterator
extends CachedSentenceIterator

Iterates through a Document, one Sentence per Example. Used for Named Entity Mentions.

Author:

Sebastian Arnold

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Annotation.Source	source

Fields inherited from class de.datexis.tagger.CachedSentenceIterator

docIt, sentIt

Fields inherited from class de.datexis.tagger.AbstractIterator

batchSize, currDocument, cursor, docsInUse, documents, encoders, inputSize, labelSize, log, name, numExamples, randomize, startTime, tagset, totalExamples

Constructor Summary

Constructors

Constructor and Description
MentionTaggerIterator(Collection<Document> docs, String name, EncoderSet encoders, Class tagset, Annotation.Source source, int numExamples, int batchSize, boolean randomize) Create an Iterator for training.
MentionTaggerIterator(Collection<Document> docs, String name, EncoderSet encoders, Class tagset, int numExamples, int batchSize, boolean randomize)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.dataset.DataSet	generateDataSet(ArrayList<Sentence> examples, int num, int exampleSize)
List<Token>	nextTokens() Returns the next Sentence in the Dataset as Tokens.

Methods inherited from class de.datexis.tagger.CachedSentenceIterator

asyncSupported, clearCache, clearCachedDocument, encodeDocument, hasNext, hasNextDocument, hasNextSentence, next, nextBatchOfSentences, nextDataSet, nextDataSet, nextDocument, nextSentence, reset

Methods inherited from class de.datexis.tagger.AbstractIterator

batch, cursor, getDocuments, getInputSize, getLabels, getLabelSize, getPreProcessor, getTagset, inputColumns, next, numExamples, randomizeDocuments, reportProgress, resetSupported, setPreProcessor, totalExamples, totalOutcomes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Field Detail

source

protected Annotation.Source source

Constructor Detail

MentionTaggerIterator

public MentionTaggerIterator(Collection<Document> docs,
                             String name,
                             EncoderSet encoders,
                             Class tagset,
                             Annotation.Source source,
                             int numExamples,
                             int batchSize,
                             boolean randomize)

Create an Iterator for training. Each example is a sentence.

Parameters:

docs - the documents to use for training.

name -

encoders - input encoders to use. Tokens will be encoded and cached on the fly.

tagset - the tagset to use for prediction.

source - the Annotation source to use for prediction

numExamples - number of examples to train. -1 for complete dataset.

batchSize - batch size in sentences

randomize - randomize examples before training epoch

MentionTaggerIterator

public MentionTaggerIterator(Collection<Document> docs,
                             String name,
                             EncoderSet encoders,
                             Class tagset,
                             int numExamples,
                             int batchSize,
                             boolean randomize)

Method Detail

nextTokens

public List<Token> nextTokens()

Returns the next Sentence in the Dataset as Tokens. These Tokens contain BOD, BOS, EOS and EOD Markers.

Returns:

generateDataSet

public org.nd4j.linalg.dataset.DataSet generateDataSet(ArrayList<Sentence> examples,
                                                       int num,
                                                       int exampleSize)

Specified by:

generateDataSet in class CachedSentenceIterator