org.apache.flink.api.java.record.io

Class ExternalProcessFixedLengthInputFormat<T extends ExternalProcessInputSplit>

java.lang.Object

org.apache.flink.api.common.io.GenericInputFormat<Record>

org.apache.flink.api.java.record.io.GenericInputFormat

org.apache.flink.api.java.record.io.ExternalProcessInputFormat<T>

org.apache.flink.api.java.record.io.ExternalProcessFixedLengthInputFormat<T>

Type Parameters:

T - The type of the input split (must extend ExternalProcessInputSplit)

All Implemented Interfaces:

Serializable, InputFormat<Record,GenericInputSplit>, InputSplitSource<GenericInputSplit>

public abstract class ExternalProcessFixedLengthInputFormat<T extends ExternalProcessInputSplit>
extends ExternalProcessInputFormat<T>

This input format starts an external process and reads its input from the standard out (stdout) of the started process. The input is split into fixed-sized segments from which a Record is generated. The external process is started outside of the JVM via a provided start command and can be an arbitrary program, e.g., a data generator or a shell script. The input format checks the exit code of the process to validate whether the process terminated correctly. A list of allowed exit codes can be provided. The input format requires (ExternalProcessInputSplit objects that hold the command to execute. Warning: This format does not consume the standard error stream (stderr) of the started process. This might cause deadlocks.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	recordLength fixed length of all records
static String	RECORDLENGTH_PARAMETER_KEY The config parameter which defines the fixed length of a record.

Fields inherited from class org.apache.flink.api.java.record.io.ExternalProcessInputFormat

allowedExitCodes, ALLOWEDEXITCODES_PARAMETER_KEY, extProcErrStream, extProcOutStream

Fields inherited from class org.apache.flink.api.common.io.GenericInputFormat

partitionNumber

Constructor Summary

Constructors

Constructor and Description
ExternalProcessFixedLengthInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration parameters)
Record	nextRecord(Record reuse)
void	open(GenericInputSplit split)
boolean	reachedEnd()
abstract boolean	readBytes(Record target, byte[] buffer, int startPos) Reads a record out of the given buffer.
void	setTargetReadBufferSize(int targetReadBufferSize) Sets the target size of the buffer to be used to read from the stdout stream.

Methods inherited from class org.apache.flink.api.java.record.io.ExternalProcessInputFormat

close, waitForProcessToFinish

Methods inherited from class org.apache.flink.api.common.io.GenericInputFormat

createInputSplits, getInputSplitAssigner, getStatistics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RECORDLENGTH_PARAMETER_KEY

public static final String RECORDLENGTH_PARAMETER_KEY

The config parameter which defines the fixed length of a record.

See Also:

Constant Field Values

recordLength

protected int recordLength

fixed length of all records

Constructor Detail

ExternalProcessFixedLengthInputFormat

public ExternalProcessFixedLengthInputFormat()

Method Detail

readBytes

public abstract boolean readBytes(Record target,
                byte[] buffer,
                int startPos)

Reads a record out of the given buffer. This operation always consumes the standard number of bytes, regardless of whether the produced record was valid.

Parameters:

target - The target Record

buffer - The buffer containing the binary data.

startPos - The start position in the byte array.

Returns:

True, is the record is valid, false otherwise.

configure

public void configure(Configuration parameters)

Specified by:

configure in interface InputFormat<Record,GenericInputSplit>

Overrides:

configure in class ExternalProcessInputFormat<T extends ExternalProcessInputSplit>

setTargetReadBufferSize

public void setTargetReadBufferSize(int targetReadBufferSize)

Sets the target size of the buffer to be used to read from the stdout stream. The actual size depends on the record length since it is chosen such that records are not split. This method has only an effect, if it is called before the input format is opened.

Parameters:

targetReadBufferSize - The target size of the read buffer.

open

public void open(GenericInputSplit split)
          throws IOException

Specified by:

open in interface InputFormat<Record,GenericInputSplit>

Overrides:

open in class ExternalProcessInputFormat<T extends ExternalProcessInputSplit>

Throws:

IOException

reachedEnd

public boolean reachedEnd()
                   throws IOException

Throws:

IOException

nextRecord

public Record nextRecord(Record reuse)
                  throws IOException

Throws:

IOException