de.datexis.sector.reader

Class Wiki727Reader

java.lang.Object

de.datexis.reader.DirectoryDatasetReader<RawTextDatasetReader>

de.datexis.reader.RawTextDatasetReader

de.datexis.sector.reader.Wiki727Reader

All Implemented Interfaces:

DatasetReader

public class Wiki727Reader
extends RawTextDatasetReader

Reader for Wiki-727l Dataset by Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018. "Text segmentation as a supervised learning task." In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 469–473.

Author:

Sebastian Arnold

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	log
protected Pattern	SECTION_PATTERN
protected int	sectionLevel
protected boolean	skipPrefaceAnnotation
protected boolean	skipPrefaceText

Fields inherited from class de.datexis.reader.RawTextDatasetReader

generateUIDs, isTokenized, useFirstSentenceAsTitle

Fields inherited from class de.datexis.reader.DirectoryDatasetReader

limit, randomizeDocuments

Constructor Summary

Constructors

Constructor and Description
Wiki727Reader()

Method Summary

Modifier and Type	Method and Description
Document	readDocumentFromFile(Resource file) Read a single Document from file.
Wiki727Reader	withSectionLevel(int level) Create Annotations down to a given level. 0 - include all section 1 - the whole document as one section 2 - include sections 3 - include subsections etc.
Wiki727Reader	withSkipPreface(boolean skip)

Methods inherited from class de.datexis.reader.RawTextDatasetReader

withFirstSentenceAsTitle, withGeneratedUIDs, withTokenizedInput

Methods inherited from class de.datexis.reader.DirectoryDatasetReader

read, readDatasetFromDirectory, readDatasetFromDirectory, stream, streamDocumentsFromDirectory, tryReadDocumentsFromFile, withLimitNumberOfDocuments, withRandomizedDocuments

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

sectionLevel

protected int sectionLevel

skipPrefaceText

protected boolean skipPrefaceText

skipPrefaceAnnotation

protected boolean skipPrefaceAnnotation

SECTION_PATTERN

protected Pattern SECTION_PATTERN

Constructor Detail

Wiki727Reader

public Wiki727Reader()

Method Detail

withSectionLevel

public Wiki727Reader withSectionLevel(int level)

Create Annotations down to a given level. 0 - include all section 1 - the whole document

as one section 2 - include

sections 3 - include

subsections etc.

withSkipPreface

public Wiki727Reader withSkipPreface(boolean skip)

readDocumentFromFile

public Document readDocumentFromFile(Resource file)

Read a single Document from file.

Overrides:

readDocumentFromFile in class RawTextDatasetReader