org.apache.flink.api.java.record.io

Class ExternalProcessInputFormat<T extends ExternalProcessInputSplit>

java.lang.Object

org.apache.flink.api.common.io.GenericInputFormat<Record>

org.apache.flink.api.java.record.io.GenericInputFormat

org.apache.flink.api.java.record.io.ExternalProcessInputFormat<T>

Type Parameters:

T - The type of the input split (must extend ExternalProcessInputSplit)

All Implemented Interfaces:

Serializable, InputFormat<Record,GenericInputSplit>, InputSplitSource<GenericInputSplit>

Direct Known Subclasses:

ExternalProcessFixedLengthInputFormat

public abstract class ExternalProcessInputFormat<T extends ExternalProcessInputSplit>
extends GenericInputFormat

This input format starts an external process and reads its input from the standard out (stdout) of the started process. The process is started outside of the JVM via a provided start command and can be an arbitrary program, e.g., a data generator or a shell script. The input format checks the exit code of the process to validate whether the process terminated correctly. A list of allowed exit codes can be provided. The input format requires (ExternalProcessInputSplit objects that hold the command to execute. Attention!
You must take care to read from (and process) both output streams of the process, standard out (stdout) and standard error (stderr). Otherwise, the input format might get deadlocked!

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allowedExitCodes Array of allowed exit codes
static String	ALLOWEDEXITCODES_PARAMETER_KEY The config parameter lists (comma separated) all allowed exit codes
protected InputStream	extProcErrStream The stderr stream of the external process
protected InputStream	extProcOutStream The stdout stream of the external process

Fields inherited from class org.apache.flink.api.common.io.GenericInputFormat

partitionNumber

Constructor Summary

Constructors

Constructor and Description
ExternalProcessInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	configure(Configuration parameters)
void	open(GenericInputSplit split)
void	waitForProcessToFinish()

Methods inherited from class org.apache.flink.api.common.io.GenericInputFormat

createInputSplits, getInputSplitAssigner, getStatistics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.io.InputFormat

nextRecord, reachedEnd

Field Detail

ALLOWEDEXITCODES_PARAMETER_KEY

public static final String ALLOWEDEXITCODES_PARAMETER_KEY

The config parameter lists (comma separated) all allowed exit codes

See Also:

Constant Field Values

extProcOutStream

protected InputStream extProcOutStream

The stdout stream of the external process

extProcErrStream

protected InputStream extProcErrStream

The stderr stream of the external process

allowedExitCodes

protected int[] allowedExitCodes

Array of allowed exit codes

Constructor Detail

ExternalProcessInputFormat

public ExternalProcessInputFormat()

Method Detail

configure

public void configure(Configuration parameters)

Specified by:

configure in interface InputFormat<Record,GenericInputSplit>

Overrides:

configure in class GenericInputFormat<Record>

close

public void close()
           throws IOException

Specified by:

close in interface InputFormat<Record,GenericInputSplit>

Overrides:

close in class GenericInputFormat<Record>

Throws:

IOException

open

public void open(GenericInputSplit split)
          throws IOException

Specified by:

open in interface InputFormat<Record,GenericInputSplit>

Overrides:

open in class GenericInputFormat<Record>

Throws:

IOException

waitForProcessToFinish

public void waitForProcessToFinish()
                            throws InterruptedException

Throws:

InterruptedException