org.apache.flink.api.common.io

Class DelimitedInputFormat<OT>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<OT>

All Implemented Interfaces:

Serializable, InputFormat<OT,FileInputSplit>, InputSplitSource<FileInputSplit>

Direct Known Subclasses:

GenericCsvInputFormat

public abstract class DelimitedInputFormat<OT>
extends FileInputFormat<OT>

Base implementation for input formats that split the input at a delimiter into records. The parsing of the record bytes into the record has to be implemented in the readRecord(Object, byte[], int, int) method.

The default delimiter is the newline character '\n'.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	DelimitedInputFormat.AbstractConfigBuilder<T> Abstract builder used to set parameters to the input format's configuration in a fluent way.
static class	DelimitedInputFormat.ConfigBuilder A builder used to set parameters to the input format's configuration in a fluent way.

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	RECORD_DELIMITER The configuration key to set the record delimiter.

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Modifier	Constructor and Description
	DelimitedInputFormat()
protected	DelimitedInputFormat(Path filePath)

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Closes the input by releasing all buffers and closing the file input stream.
void	configure(Configuration parameters) Configures this input format by reading the path to the file from the configuration andge the string that defines the record delimiter.
static DelimitedInputFormat.ConfigBuilder	configureDelimitedFormat(FileDataSourceBase<?> target) Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.
int	getBufferSize()
byte[]	getDelimiter()
int	getLineLengthLimit()
int	getNumLineSamples()
FileInputFormat.FileBaseStatistics	getStatistics(BaseStatistics cachedStats) Obtains basic file statistics containing only file size.
protected static void	loadGloablConfigParams()
OT	nextRecord(OT record) Tries to read the next pair from the input.
void	open(FileInputSplit split) Opens the given input split.
boolean	reachedEnd() Checks whether the current split is at its end.
protected boolean	readLine()
abstract OT	readRecord(OT reuse, byte[] bytes, int offset, int numBytes) This function parses the given byte array which represents a serialized record.
void	setBufferSize(int bufferSize)
void	setDelimiter(byte[] delimiter)
void	setDelimiter(char delimiter)
void	setDelimiter(String delimiter)
void	setLineLengthLimit(int lineLengthLimit)
void	setNumLineSamples(int numLineSamples)

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, configureFileFormat, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

RECORD_DELIMITER

protected static final String RECORD_DELIMITER

The configuration key to set the record delimiter.

See Also:

Constant Field Values

Constructor Detail

DelimitedInputFormat

public DelimitedInputFormat()

DelimitedInputFormat

protected DelimitedInputFormat(Path filePath)

Method Detail

loadGloablConfigParams

protected static void loadGloablConfigParams()

getDelimiter

public byte[] getDelimiter()

setDelimiter

public void setDelimiter(byte[] delimiter)

setDelimiter

public void setDelimiter(char delimiter)

setDelimiter

public void setDelimiter(String delimiter)

getLineLengthLimit

public int getLineLengthLimit()

setLineLengthLimit

public void setLineLengthLimit(int lineLengthLimit)

getBufferSize

public int getBufferSize()

setBufferSize

public void setBufferSize(int bufferSize)

getNumLineSamples

public int getNumLineSamples()

setNumLineSamples

public void setNumLineSamples(int numLineSamples)

readRecord

public abstract OT readRecord(OT reuse,
            byte[] bytes,
            int offset,
            int numBytes)
                       throws IOException

This function parses the given byte array which represents a serialized record. The function returns a valid record or throws an IOException.

Parameters:

reuse - An optionally reusable object.

bytes - Binary data of serialized records.

offset - The offset where to start to read the record data.

numBytes - The number of bytes that can be read starting at the offset position.

Returns:

Returns the read record if it was successfully deserialized.

Throws:

IOException - if the record could not be read.

configure

public void configure(Configuration parameters)

Configures this input format by reading the path to the file from the configuration andge the string that defines the record delimiter.

Specified by:

configure in interface InputFormat<OT,FileInputSplit>

Overrides:

configure in class FileInputFormat<OT>

Parameters:

parameters - The configuration object to read the parameters from.

See Also:

InputFormat.configure(org.apache.flink.configuration.Configuration)

getStatistics

public FileInputFormat.FileBaseStatistics getStatistics(BaseStatistics cachedStats)
                                                 throws IOException

Description copied from class: FileInputFormat

Obtains basic file statistics containing only file size. If the input is a directory, then the size is the sum of all contained files.

Specified by:

getStatistics in interface InputFormat<OT,FileInputSplit>

Overrides:

getStatistics in class FileInputFormat<OT>

Parameters:

cachedStats - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

Throws:

IOException

See Also:

InputFormat.getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics)

open

public void open(FileInputSplit split)
          throws IOException

Opens the given input split. This method opens the input stream to the specified file, allocates read buffers and positions the stream at the correct position, making sure that any partial record at the beginning is skipped.

Specified by:

open in interface InputFormat<OT,FileInputSplit>

Overrides:

open in class FileInputFormat<OT>

Parameters:

split - The input split to open.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

See Also:

FileInputFormat.open(org.apache.flink.core.fs.FileInputSplit)

reachedEnd

public boolean reachedEnd()

Checks whether the current split is at its end.

Returns:

True, if the split is at its end, false otherwise.

nextRecord

public OT nextRecord(OT record)
              throws IOException

Description copied from interface: InputFormat

Tries to read the next pair from the input. By using the return value invalid records in the input can be skipped.

When this method is called, the input format it guaranteed to be opened.

Parameters:

record - Object that may be reused.

Returns:

Indicates whether the record could be successfully read. A return value of true indicates that the read was successful, a return value of false indicates that the current record was not read successfully and should be skipped.

Throws:

IOException - Thrown, if an I/O error occurred.

close

public void close()
           throws IOException

Closes the input by releasing all buffers and closing the file input stream.

Specified by:

close in interface InputFormat<OT,FileInputSplit>

Overrides:

close in class FileInputFormat<OT>

Throws:

IOException - Thrown, if the closing of the file stream causes an I/O error.

readLine

protected final boolean readLine()
                          throws IOException

Throws:

IOException

configureDelimitedFormat

public static DelimitedInputFormat.ConfigBuilder configureDelimitedFormat(FileDataSourceBase<?> target)

Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.

Returns:

A config builder for setting parameters.