org.apache.flink.api.common.io

Class BinaryInputFormat<T>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<T>

org.apache.flink.api.common.io.BinaryInputFormat<T>

All Implemented Interfaces:

Serializable, InputFormat<T,FileInputSplit>, InputSplitSource<FileInputSplit>

Direct Known Subclasses:

SerializedInputFormat

public abstract class BinaryInputFormat<T>
extends FileInputFormat<T>

Base class for all input formats that use blocks of fixed size. The input splits are aligned to these blocks. Without configuration, these block sizes equal the native block sizes of the HDFS.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	BinaryInputFormat.BlockBasedInput Writes a block info at the end of the blocks. Current implementation uses only int and not long.

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.AbstractConfigBuilder<T>, FileInputFormat.ConfigBuilder, FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static String	BLOCK_SIZE_PARAMETER_KEY The config parameter which defines the fixed length of a record.
static long	NATIVE_BLOCK_SIZE

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
BinaryInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration parameters) Configures the file input format by reading the file path from the configuration.
BlockInfo	createBlockInfo()
FileInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits for the file.
protected org.apache.flink.api.common.io.BinaryInputFormat.SequentialStatistics	createStatistics(List<FileStatus> files, FileInputFormat.FileBaseStatistics stats) Fill in the statistics.
protected abstract T	deserialize(T reuse, DataInputView dataInput)
protected List<FileStatus>	getFiles()
protected FileInputSplit[]	getInputSplits()
org.apache.flink.api.common.io.BinaryInputFormat.SequentialStatistics	getStatistics(BaseStatistics cachedStats) Obtains basic file statistics containing only file size.
T	nextRecord(T record) Tries to read the next pair from the input.
void	open(FileInputSplit split) Opens an input stream to the file defined in the input format.
boolean	reachedEnd() Method used to check if the end of the input is reached.

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, close, configureFileFormat, decorateInputStream, extractFileExtension, getFilePath, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

BLOCK_SIZE_PARAMETER_KEY

public static final String BLOCK_SIZE_PARAMETER_KEY

The config parameter which defines the fixed length of a record.

See Also:

Constant Field Values

NATIVE_BLOCK_SIZE

public static final long NATIVE_BLOCK_SIZE

See Also:

Constant Field Values

Constructor Detail

BinaryInputFormat

public BinaryInputFormat()

Method Detail

configure

public void configure(Configuration parameters)

Description copied from class: FileInputFormat

Configures the file input format by reading the file path from the configuration.

Specified by:

configure in interface InputFormat<T,FileInputSplit>

Overrides:

configure in class FileInputFormat<T>

Parameters:

parameters - The configuration with all parameters.

See Also:

InputFormat.configure(org.apache.flink.configuration.Configuration)

createInputSplits

public FileInputSplit[] createInputSplits(int minNumSplits)
                                   throws IOException

Description copied from class: FileInputFormat

Computes the input splits for the file. By default, one file block is one split. If more splits are requested than blocks are available, then a split may be a fraction of a block and splits may cross block boundaries.

Specified by:

createInputSplits in interface InputFormat<T,FileInputSplit>

Specified by:

createInputSplits in interface InputSplitSource<FileInputSplit>

Overrides:

createInputSplits in class FileInputFormat<T>

Parameters:

minNumSplits - The minimum desired number of file splits.

Returns:

The computed file splits.

Throws:

IOException - Thrown, when the creation of the splits was erroneous.

See Also:

InputFormat.createInputSplits(int)

getFiles

protected List<FileStatus> getFiles()
                             throws IOException

Throws:

IOException

getStatistics

public org.apache.flink.api.common.io.BinaryInputFormat.SequentialStatistics getStatistics(BaseStatistics cachedStats)

Description copied from class: FileInputFormat

Obtains basic file statistics containing only file size. If the input is a directory, then the size is the sum of all contained files.

Specified by:

getStatistics in interface InputFormat<T,FileInputSplit>

Overrides:

getStatistics in class FileInputFormat<T>

Parameters:

cachedStats - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

See Also:

InputFormat.getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics)

getInputSplits

protected FileInputSplit[] getInputSplits()
                                   throws IOException

Throws:

IOException

createBlockInfo

public BlockInfo createBlockInfo()

createStatistics

protected org.apache.flink.api.common.io.BinaryInputFormat.SequentialStatistics createStatistics(List<FileStatus> files,
                                                                                     FileInputFormat.FileBaseStatistics stats)
                                                                                          throws IOException

Fill in the statistics. The last modification time and the total input size are prefilled.

Parameters:

files - The files that are associated with this block input format.

stats - The pre-filled statistics.

Throws:

IOException

open

public void open(FileInputSplit split)
          throws IOException

Description copied from class: FileInputFormat

Opens an input stream to the file defined in the input format. The stream is positioned at the beginning of the given split.

The stream is actually opened in an asynchronous thread to make sure any interruptions to the thread working on the input format do not reach the file system.

Specified by:

open in interface InputFormat<T,FileInputSplit>

Overrides:

open in class FileInputFormat<T>

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

reachedEnd

public boolean reachedEnd()
                   throws IOException

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

nextRecord

public T nextRecord(T record)
             throws IOException

Description copied from interface: InputFormat

Tries to read the next pair from the input. By using the return value invalid records in the input can be skipped.

When this method is called, the input format it guaranteed to be opened.

Parameters:

record - Object that may be reused.

Returns:

Indicates whether the record could be successfully read. A return value of true indicates that the read was successful, a return value of false indicates that the current record was not read successfully and should be skipped.

Throws:

IOException - Thrown, if an I/O error occurred.

deserialize

protected abstract T deserialize(T reuse,
            DataInputView dataInput)
                          throws IOException

Throws:

IOException