org.apache.flink.api.common.io

Interface InputFormat<OT,T extends InputSplit>

Type Parameters:

OT - The type of the produced records.

T - The type of input split.

All Superinterfaces:

InputSplitSource<T>, Serializable

All Known Implementing Classes:

BinaryInputFormat, DelimitedInputFormat, FileInputFormat, GenericCsvInputFormat, GenericInputFormat, ReplicatingInputFormat, RichInputFormat, SerializedInputFormat

@Public
public interface InputFormat<OT,T extends InputSplit>
extends InputSplitSource<T>, Serializable

The base interface for data sources that produces records.

The input format handles the following:

• It describes how the input is split into splits that can be processed in parallel.
• It describes how to read records from the input split.
• It describes how to gather basic statistics from the input.

The life cycle of an input format is the following:

1. After being instantiated (parameterless), it is configured with a Configuration object. Basic fields are read from the configuration, such as for example a file path, if the format describes files as input.
2. Optionally: It is called by the compiler to produce basic statistics about the input.
3. It is called to create the input splits.
4. Each parallel input task creates an instance, configures it and opens it for a specific split.
5. All records are read from the input
6. The input format is closed

IMPORTANT NOTE: Input formats must be written such that an instance can be opened again after it was closed. That is due to the fact that the input format is used for potentially multiple splits. After a split is done, the format's close function is invoked and, if another split is available, the open function is invoked afterwards for the next split.

See Also:

InputSplit, BaseStatistics

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Method that marks the end of the life-cycle of an input split.
void	configure(Configuration parameters) Configures this input format.
T[]	createInputSplits(int minNumSplits) Creates the different splits of the input that can be processed in parallel.
InputSplitAssigner	getInputSplitAssigner(T[] inputSplits) Gets the type of the input splits that are processed by this input format.
BaseStatistics	getStatistics(BaseStatistics cachedStatistics) Gets the basic statistics from the input described by this format.
OT	nextRecord(OT reuse) Reads the next record from the input.
void	open(T split) Opens a parallel instance of the input format to work on a split.
boolean	reachedEnd() Method used to check if the end of the input is reached.

Method Detail

configure

void configure(Configuration parameters)

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

getStatistics

BaseStatistics getStatistics(BaseStatistics cachedStatistics)
                             throws IOException

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format it guaranteed to be configured.

Parameters:

cachedStatistics - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

Throws:

IOException

createInputSplits

T[] createInputSplits(int minNumSplits)
                                         throws IOException

Creates the different splits of the input that can be processed in parallel.

When this method is called, the input format it guaranteed to be configured.

Specified by:

createInputSplits in interface InputSplitSource<T extends InputSplit>

Parameters:

minNumSplits - The minimum desired number of splits. If fewer are created, some parallel instances may remain idle.

Returns:

The splits of this input that can be processed in parallel.

Throws:

IOException - Thrown, when the creation of the splits was erroneous.

getInputSplitAssigner

InputSplitAssigner getInputSplitAssigner(T[] inputSplits)

Gets the type of the input splits that are processed by this input format.

Specified by:

getInputSplitAssigner in interface InputSplitSource<T extends InputSplit>

Returns:

The type of the input splits.

open

void open(T split)
          throws IOException

Opens a parallel instance of the input format to work on a split.

When this method is called, the input format it guaranteed to be configured.

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

reachedEnd

boolean reachedEnd()
                   throws IOException

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

nextRecord

OT nextRecord(OT reuse)
              throws IOException

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Parameters:

reuse - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

close

void close()
           throws IOException

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Throws:

IOException - Thrown, if the input could not be closed properly.