Package net.sf.okapi.steps.tokenization

Class TokenizationStep

java.lang.Object

net.sf.okapi.common.pipeline.BasePipelineStep

net.sf.okapi.steps.tokenization.TokenizationStep

All Implemented Interfaces:

AutoCloseable, Function<Stream<Event>,Stream<Event>>, IPipelineStep

public class TokenizationStep
extends BasePipelineStep

Constructor Summary

Constructors

Constructor	Description
TokenizationStep()

Method Summary

Modifier and Type	Method	Description
List<Token>	apostrophe(Token token, LocaleId locale)	Break French and Italian words with apostrophe into three tokens WORD, PUNCTUATION, WORD
String	getDescription()
String	getName()
LocaleId	getSourceLocale()
LocaleId	getTargetLocale()
protected Event	handleStartDocument(Event event)
protected Event	handleTextUnit(Event event)
Collection<? extends Token>	postProcess(Token t, LocaleId language)	Various rules to make corrections to RbbiTokenizer
void	setSourceLocale(LocaleId sourceLocale)
void	setTargetLocale(LocaleId targetLocale)

Methods inherited from class net.sf.okapi.common.pipeline.BasePipelineStep

cancel, destroy, getHelpLocation, getParameters, handleCustom, handleDocumentPart, handleEndBatch, handleEndBatchItem, handleEndDocument, handleEndGroup, handleEndSubDocument, handleEndSubfilter, handleEvent, handleMultiEvent, handlePipelineParameters, handleRawDocument, handleStartBatch, handleStartBatchItem, handleStartGroup, handleStartSubDocument, handleStartSubfilter, isDone, isLastOutputStep, setLastOutputStep, setParameters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Function

andThen, compose

Methods inherited from interface net.sf.okapi.common.pipeline.IPipelineStep

apply, close, handleStream

Constructor Detail

TokenizationStep

public TokenizationStep()

Method Detail

handleStartDocument

protected Event handleStartDocument(Event event)

Overrides:

handleStartDocument in class BasePipelineStep

handleTextUnit

protected Event handleTextUnit(Event event)

Overrides:

handleTextUnit in class BasePipelineStep

getSourceLocale

public LocaleId getSourceLocale()

Specified by:

getSourceLocale in interface IPipelineStep

Overrides:

getSourceLocale in class BasePipelineStep

setSourceLocale

public void setSourceLocale(LocaleId sourceLocale)

Specified by:

setSourceLocale in interface IPipelineStep

Overrides:

setSourceLocale in class BasePipelineStep

getTargetLocale

public LocaleId getTargetLocale()

Specified by:

getTargetLocale in interface IPipelineStep

Overrides:

getTargetLocale in class BasePipelineStep

setTargetLocale

public void setTargetLocale(LocaleId targetLocale)

Specified by:

setTargetLocale in interface IPipelineStep

Overrides:

setTargetLocale in class BasePipelineStep

postProcess

public Collection<? extends Token> postProcess(Token t,
                                               LocaleId language)

Various rules to make corrections to RbbiTokenizer

Parameters:

t - the Token

Returns:

list of correct tokens or the original token if no changes were made

apostrophe

public List<Token> apostrophe(Token token,
                              LocaleId locale)

Break French and Italian words with apostrophe into three tokens WORD, PUNCTUATION, WORD

Parameters:

token -

Returns:

list of transformed tokens if any

getName

public String getName()

getDescription

public String getDescription()