org.apache.flink.streaming.api.environment

Class StreamExecutionEnvironment

java.lang.Object

org.apache.flink.streaming.api.environment.StreamExecutionEnvironment

Direct Known Subclasses:

LocalStreamEnvironment, RemoteStreamEnvironment, StreamContextEnvironment, StreamPlanEnvironment

public abstract class StreamExecutionEnvironment
extends Object

ExecutionEnvironment for streaming jobs. An instance of it is necessary to construct streaming topologies.

Field Summary

Fields

Modifier and Type	Field and Description
protected static StreamExecutionEnvironment	currentEnvironment
static String	DEFAULT_JOB_NAME
protected StreamGraph	streamGraph

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	StreamExecutionEnvironment() Constructor for creating StreamExecutionEnvironment

Method Summary

Modifier and Type	Method and Description
void	addDefaultKryoSerializer(Class<?> type, Class<? extends com.esotericsoftware.kryo.Serializer<?>> serializerClass) Adds a new Kryo default serializer to the Runtime.
void	addDefaultKryoSerializer(Class<?> type, com.esotericsoftware.kryo.Serializer<?> serializer) Adds a new Kryo default serializer to the Runtime.
<OUT> DataStreamSource<OUT>	addSource(SourceFunction<OUT> function) Adds a data source with a custom type information thus opening a DataStream.
<OUT> DataStreamSource<OUT>	addSource(SourceFunction<OUT> function, String sourceName) Ads a data source with a custom type information thus opening a DataStream.
<F> F	clean(F f) Returns a "closure-cleaned" version of the given function.
<OUT> DataStreamSource<OUT>	createInput(org.apache.flink.api.common.io.InputFormat<OUT,?> inputFormat) Generic method to create an input data stream with InputFormat.
<OUT> DataStreamSource<OUT>	createInput(org.apache.flink.api.common.io.InputFormat<OUT,?> inputFormat, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo) Generic method to create an input data stream with InputFormat.
static LocalStreamEnvironment	createLocalEnvironment() Creates a LocalStreamEnvironment.
static LocalStreamEnvironment	createLocalEnvironment(int parallelism) Creates a LocalStreamEnvironment.
static StreamExecutionEnvironment	createRemoteEnvironment(String host, int port, int parallelism, String... jarFiles) Creates a RemoteStreamEnvironment.
static StreamExecutionEnvironment	createRemoteEnvironment(String host, int port, String... jarFiles) Creates a RemoteStreamEnvironment.
StreamExecutionEnvironment	disableOperatorChaining() Disables operator chaining for streaming operators.
StreamExecutionEnvironment	enableCheckpointing() Method for enabling fault-tolerance.
StreamExecutionEnvironment	enableCheckpointing(long interval) Method for enabling fault-tolerance.
StreamExecutionEnvironment	enableCheckpointing(long interval, boolean force) Deprecated.
abstract org.apache.flink.api.common.JobExecutionResult	execute() Triggers the program execution.
abstract org.apache.flink.api.common.JobExecutionResult	execute(String jobName) Triggers the program execution.
<OUT> DataStreamSource<OUT>	fromCollection(Collection<OUT> data) Creates a data stream from the given non-empty collection.
<OUT> DataStreamSource<OUT>	fromCollection(Collection<OUT> data, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo) Creates a data stream from the given non-empty collection.Note that this operation will result in a non-parallel data stream source, i.e.
<OUT> DataStreamSource<OUT>	fromCollection(Iterator<OUT> data, Class<OUT> type) Creates a data stream from the given iterator.
<OUT> DataStreamSource<OUT>	fromCollection(Iterator<OUT> data, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo) Creates a data stream from the given iterator.
<OUT> DataStreamSource<OUT>	fromElements(OUT... data) Creates a new data stream that contains the given elements.
<OUT> DataStreamSource<OUT>	fromParallelCollection(org.apache.flink.util.SplittableIterator<OUT> iterator, Class<OUT> type) Creates a new data stream that contains elements in the iterator.
<OUT> DataStreamSource<OUT>	fromParallelCollection(org.apache.flink.util.SplittableIterator<OUT> iterator, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo) Creates a new data stream that contains elements in the iterator.
DataStreamSource<Long>	generateParallelSequence(long from, long to) Creates a new data stream that contains a sequence of numbers.
DataStreamSource<Long>	generateSequence(long from, long to) Creates a new data stream that contains a sequence of numbers.
long	getBufferTimeout() Sets the maximum time frequency (milliseconds) for the flushing of the output buffers.
org.apache.flink.api.common.ExecutionConfig	getConfig() Gets the config object.
int	getDegreeOfParallelism() Deprecated. Please use getParallelism()
static StreamExecutionEnvironment	getExecutionEnvironment() Creates an execution environment that represents the context in which the program is currently executed.
String	getExecutionPlan() Creates the plan with which the system will execute the program, and returns it as a String using a JSON representation of the execution data flow graph.
int	getNumberOfExecutionRetries() Gets the number of times the system will try to re-execute failed tasks.
int	getParallelism() Gets the parallelism with which operation are executed by default.
StreamGraph	getStreamGraph() Getter of the StreamGraph of the streaming job.
protected static void	initializeFromFactory(StreamExecutionEnvironmentFactory eef)
<OUT> DataStreamSource<OUT>	readFile(org.apache.flink.api.common.io.FileInputFormat<OUT> inputFormat, String filePath) Reads the given file with the given imput format.
<OUT> DataStreamSource<OUT>	readFileOfPrimitives(String filePath, Class<OUT> typeClass) Creates a data stream that represents the primitive type produced by reading the given file line wise.
<OUT> DataStreamSource<OUT>	readFileOfPrimitives(String filePath, String delimiter, Class<OUT> typeClass) Creates a data stream that represents the primitive type produced by reading the given file in delimited way.
DataStream<String>	readFileStream(String filePath, long intervalMillis, FileMonitoringFunction.WatchType watchType) Creates a data stream that contains the contents of file created while system watches the given path.
DataStreamSource<String>	readTextFile(String filePath) Creates a data stream that represents the Strings produced by reading the given file line wise.
DataStreamSource<String>	readTextFile(String filePath, String charsetName) Creates a data stream that represents the Strings produced by reading the given file line wise.
DataStreamSource<org.apache.flink.types.StringValue>	readTextFileWithValue(String filePath) Creates a data stream that represents the strings produced by reading the given file line wise.
DataStreamSource<org.apache.flink.types.StringValue>	readTextFileWithValue(String filePath, String charsetName, boolean skipInvalidLines) Creates a data stream that represents the Strings produced by reading the given file line wise.
void	registerType(Class<?> type) Registers the given type with the serialization stack.
void	registerTypeWithKryoSerializer(Class<?> type, Class<? extends com.esotericsoftware.kryo.Serializer<?>> serializerClass) Registers the given Serializer via its class as a serializer for the given type at the KryoSerializer
void	registerTypeWithKryoSerializer(Class<?> type, com.esotericsoftware.kryo.Serializer<?> serializer) Registers the given type with a Kryo Serializer.
StreamExecutionEnvironment	setBufferTimeout(long timeoutMillis) Sets the maximum time frequency (milliseconds) for the flushing of the output buffers.
static void	setDefaultLocalParallelism(int parallelism) Sets the default parallelism that will be used for the local execution environment created by createLocalEnvironment().
StreamExecutionEnvironment	setDegreeOfParallelism(int parallelism) Deprecated. Please use setParallelism(int)
void	setNumberOfExecutionRetries(int numberOfExecutionRetries) Sets the number of times that failed tasks are re-executed.
StreamExecutionEnvironment	setParallelism(int parallelism) Sets the parallelism for operations executed through this environment.
StreamExecutionEnvironment	setStateHandleProvider(org.apache.flink.runtime.state.StateHandleProvider<?> provider) Sets the StateHandleProvider used for storing operator state checkpoints when checkpointing is enabled.
DataStreamSource<String>	socketTextStream(String hostname, int port) Creates a new data stream that contains the strings received infinitely from a socket.
DataStreamSource<String>	socketTextStream(String hostname, int port, char delimiter) Creates a new data stream that contains the strings received infinitely from a socket.
DataStreamSource<String>	socketTextStream(String hostname, int port, char delimiter, long maxRetry) Creates a new data stream that contains the strings received infinitely from a socket.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_JOB_NAME

public static final String DEFAULT_JOB_NAME

See Also:

Constant Field Values

currentEnvironment

protected static StreamExecutionEnvironment currentEnvironment

streamGraph

protected StreamGraph streamGraph

Constructor Detail

StreamExecutionEnvironment

protected StreamExecutionEnvironment()

Constructor for creating StreamExecutionEnvironment

Method Detail

getConfig

public org.apache.flink.api.common.ExecutionConfig getConfig()

Gets the config object.

setDegreeOfParallelism

@Deprecated
public StreamExecutionEnvironment setDegreeOfParallelism(int parallelism)

Deprecated. Please use setParallelism(int)

Sets the parallelism for operations executed through this environment. Setting a parallelism of x here will cause all operators (such as map, batchReduce) to run with x parallel instances. This method overrides the default parallelism for this environment. The LocalStreamEnvironment uses by default a value equal to the number of hardware contexts (CPU cores / threads). When executing the program via the command line client from a JAR file, the default degree of parallelism is the one configured for that setup.

Parameters:

parallelism - The parallelism

getDegreeOfParallelism

@Deprecated
public int getDegreeOfParallelism()

Deprecated. Please use getParallelism()

Gets the parallelism with which operation are executed by default. Operations can individually override this value to use a specific parallelism.

Returns:

The parallelism used by operations, unless they override that value.

setParallelism

public StreamExecutionEnvironment setParallelism(int parallelism)

Sets the parallelism for operations executed through this environment. Setting a parallelism of x here will cause all operators (such as map, batchReduce) to run with x parallel instances. This method overrides the default parallelism for this environment. The LocalStreamEnvironment uses by default a value equal to the number of hardware contexts (CPU cores / threads). When executing the program via the command line client from a JAR file, the default degree of parallelism is the one configured for that setup.

Parameters:

parallelism - The parallelism

getParallelism

public int getParallelism()

Gets the parallelism with which operation are executed by default. Operations can individually override this value to use a specific parallelism.

Returns:

The parallelism used by operations, unless they override that value.

setBufferTimeout

public StreamExecutionEnvironment setBufferTimeout(long timeoutMillis)

Sets the maximum time frequency (milliseconds) for the flushing of the output buffers. By default the output buffers flush frequently to provide low latency and to aid smooth developer experience. Setting the parameter can result in three logical modes:

• A positive integer triggers flushing periodically by that integer
• 0 triggers flushing after every record thus minimizing latency
• -1 triggers flushing only when the output buffer is full thus maximizing throughput

Parameters:

timeoutMillis - The maximum time between two output flushes.

getBufferTimeout

public long getBufferTimeout()

Sets the maximum time frequency (milliseconds) for the flushing of the output buffers. For clarification on the extremal values see setBufferTimeout(long).

Returns:

The timeout of the buffer.

disableOperatorChaining

public StreamExecutionEnvironment disableOperatorChaining()

Disables operator chaining for streaming operators. Operator chaining allows non-shuffle operations to be co-located in the same thread fully avoiding serialization and de-serialization.

Returns:

StreamExecutionEnvironment with chaining disabled.

enableCheckpointing

public StreamExecutionEnvironment enableCheckpointing(long interval)

Method for enabling fault-tolerance. Activates monitoring and backup of streaming operator states.

Setting this option assumes that the job is used in production and thus if not stated explicitly otherwise with calling with the setNumberOfExecutionRetries(int numberOfExecutionRetries) method in case of failure the job will be resubmitted to the cluster indefinitely.

Parameters:

interval - Time interval between state checkpoints in millis

enableCheckpointing

@Deprecated
public StreamExecutionEnvironment enableCheckpointing(long interval,
                                                                  boolean force)

Deprecated.

Method for force-enabling fault-tolerance. Activates monitoring and backup of streaming operator states even for jobs containing iterations. Please note that the checkpoint/restore guarantees for iterative jobs are only best-effort at the moment. Records inside the loops may be lost during failure.

Setting this option assumes that the job is used in production and thus if not stated explicitly otherwise with calling with the setNumberOfExecutionRetries(int numberOfExecutionRetries) method in case of failure the job will be resubmitted to the cluster indefinitely.

Parameters:

interval - Time interval between state checkpoints in millis

force - If true checkpointing will be enabled for iterative jobs as well

enableCheckpointing

public StreamExecutionEnvironment enableCheckpointing()

Method for enabling fault-tolerance. Activates monitoring and backup of streaming operator states.

Setting this option assumes that the job is used in production and thus if not stated explicitly otherwise with calling with the setNumberOfExecutionRetries(int numberOfExecutionRetries) method in case of failure the job will be resubmitted to the cluster indefinitely.

setStateHandleProvider

public StreamExecutionEnvironment setStateHandleProvider(org.apache.flink.runtime.state.StateHandleProvider<?> provider)

Sets the StateHandleProvider used for storing operator state checkpoints when checkpointing is enabled.

An example would be using a FileStateHandle.createProvider(String) to use any Flink supported file system as a state backend

setNumberOfExecutionRetries

public void setNumberOfExecutionRetries(int numberOfExecutionRetries)

Sets the number of times that failed tasks are re-executed. A value of zero effectively disables fault tolerance. A value of -1 indicates that the system default value (as defined in the configuration) should be used.

Parameters:

numberOfExecutionRetries - The number of times the system will try to re-execute failed tasks.

getNumberOfExecutionRetries

public int getNumberOfExecutionRetries()

Gets the number of times the system will try to re-execute failed tasks. A value of -1 indicates that the system default value (as defined in the configuration) should be used.

Returns:

The number of times the system will try to re-execute failed tasks.

setDefaultLocalParallelism

public static void setDefaultLocalParallelism(int parallelism)

Sets the default parallelism that will be used for the local execution environment created by createLocalEnvironment().

Parameters:

parallelism - The parallelism to use as the default local parallelism.

addDefaultKryoSerializer

public void addDefaultKryoSerializer(Class<?> type,
                                     com.esotericsoftware.kryo.Serializer<?> serializer)

Adds a new Kryo default serializer to the Runtime.

Note that the serializer instance must be serializable (as defined by java.io.Serializable), because it may be distributed to the worker nodes by java serialization.

Parameters:

type - The class of the types serialized with the given serializer.

serializer - The serializer to use.

addDefaultKryoSerializer

public void addDefaultKryoSerializer(Class<?> type,
                                     Class<? extends com.esotericsoftware.kryo.Serializer<?>> serializerClass)

Adds a new Kryo default serializer to the Runtime.

Parameters:

type - The class of the types serialized with the given serializer.

serializerClass - The class of the serializer to use.

registerTypeWithKryoSerializer

public void registerTypeWithKryoSerializer(Class<?> type,
                                           com.esotericsoftware.kryo.Serializer<?> serializer)

Registers the given type with a Kryo Serializer.

Note that the serializer instance must be serializable (as defined by java.io.Serializable), because it may be distributed to the worker nodes by java serialization.

Parameters:

type - The class of the types serialized with the given serializer.

serializer - The serializer to use.

registerTypeWithKryoSerializer

public void registerTypeWithKryoSerializer(Class<?> type,
                                           Class<? extends com.esotericsoftware.kryo.Serializer<?>> serializerClass)

Registers the given Serializer via its class as a serializer for the given type at the KryoSerializer

Parameters:

type - The class of the types serialized with the given serializer.

serializerClass - The class of the serializer to use.

registerType

public void registerType(Class<?> type)

Registers the given type with the serialization stack. If the type is eventually serialized as a POJO, then the type is registered with the POJO serializer. If the type ends up being serialized with Kryo, then it will be registered at Kryo to make sure that only tags are written.

Parameters:

type - The class of the type to register.

generateSequence

public DataStreamSource<Long> generateSequence(long from,
                                               long to)

Creates a new data stream that contains a sequence of numbers. The data stream will be created with parallelism one, so the order of the elements is guaranteed.

Parameters:

from - The number to start at (inclusive)

to - The number to stop at (inclusive)

Returns:

A data stream, containing all number in the [from, to] interval

generateParallelSequence

public DataStreamSource<Long> generateParallelSequence(long from,
                                                       long to)

Creates a new data stream that contains a sequence of numbers. The data stream will be created in parallel, so there is no guarantee about the oder of the elements.

Parameters:

from - The number to start at (inclusive)

to - The number to stop at (inclusive)

Returns:

A data stream, containing all number in the [from, to] interval

fromElements

public <OUT> DataStreamSource<OUT> fromElements(OUT... data)

Creates a new data stream that contains the given elements. The elements must all be of the same type, for example, all of the String or Integer.

The framework will try and determine the exact type from the elements. In case of generic elements, it may be necessary to manually supply the type information via fromCollection(java.util.Collection, org.apache.flink.api.common.typeinfo.TypeInformation).

Note that this operation will result in a non-parallel data stream source, i.e. a data stream source with a degree of parallelism one.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

data - The array of elements to create the data stream from.

Returns:

The data stream representing the given array of elements

fromCollection

public <OUT> DataStreamSource<OUT> fromCollection(Collection<OUT> data)

Creates a data stream from the given non-empty collection. The type of the data stream is that of the elements in the collection.

The framework will try and determine the exact type from the collection elements. In case of generic elements, it may be necessary to manually supply the type information via fromCollection(java.util.Collection, org.apache.flink.api.common.typeinfo.TypeInformation).

Note that this operation will result in a non-parallel data stream source, i.e. a data stream source with a degree of parallelism one.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

data - The collection of elements to create the data stream from

Returns:

The data stream representing the given collection

fromCollection

public <OUT> DataStreamSource<OUT> fromCollection(Collection<OUT> data,
                                                  org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo)

Creates a data stream from the given non-empty collection.Note that this operation will result in a non-parallel data stream source, i.e. a data stream source with a degree of parallelism one.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

data - The collection of elements to create the data stream from

typeInfo - The TypeInformation for the produced data stream

Returns:

The data stream representing the given collection

fromCollection

public <OUT> DataStreamSource<OUT> fromCollection(Iterator<OUT> data,
                                                  Class<OUT> type)

Creates a data stream from the given iterator. Because the iterator will remain unmodified until the actual execution happens, the type of data returned by the iterator must be given explicitly in the form of the type class (this is due to the fact that the Java compiler erases the generic type information).

Note that this operation will result in a non-parallel data stream source, i.e. a data stream source with a degree of parallelism of one.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

data - The iterator of elements to create the data stream from

type - The class of the data produced by the iterator. Must not be a generic class.

Returns:

The data stream representing the elements in the iterator

See Also:

fromCollection(java.util.Iterator, org.apache.flink.api.common.typeinfo.TypeInformation)

fromCollection

public <OUT> DataStreamSource<OUT> fromCollection(Iterator<OUT> data,
                                                  org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo)

Creates a data stream from the given iterator. Because the iterator will remain unmodified until the actual execution happens, the type of data returned by the iterator must be given explicitly in the form of the type information. This method is useful for cases where the type is generic. In that case, the type class (as given in fromCollection(java.util.Iterator, Class) does not supply all type information.

Note that this operation will result in a non-parallel data stream source, i.e. a data stream source with a degree of parallelism one.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

data - The iterator of elements to create the data stream from

typeInfo - The TypeInformation for the produced data stream

Returns:

The data stream representing the elements in the iterator

fromParallelCollection

public <OUT> DataStreamSource<OUT> fromParallelCollection(org.apache.flink.util.SplittableIterator<OUT> iterator,
                                                          Class<OUT> type)

Creates a new data stream that contains elements in the iterator. The iterator is splittable, allowing the framework to create a parallel data stream source that returns the elements in the iterator.

Because the iterator will remain unmodified until the actual execution happens, the type of data returned by the iterator must be given explicitly in the form of the type class (this is due to the fact that the Java compiler erases the generic type information).

Type Parameters:

OUT - The type of the returned data stream

Parameters:

iterator - The iterator that produces the elements of the data stream

type - The class of the data produced by the iterator. Must not be a generic class.

Returns:

A data stream representing the elements in the iterator

fromParallelCollection

public <OUT> DataStreamSource<OUT> fromParallelCollection(org.apache.flink.util.SplittableIterator<OUT> iterator,
                                                          org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo)

Creates a new data stream that contains elements in the iterator. The iterator is splittable, allowing the framework to create a parallel data stream source that returns the elements in the iterator.

Because the iterator will remain unmodified until the actual execution happens, the type of data returned by the iterator must be given explicitly in the form of the type information. This method is useful for cases where the type is generic. In that case, the type class (as given in fromParallelCollection(org.apache.flink.util.SplittableIterator, Class) does not supply all type information.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

iterator - The iterator that produces the elements of the data stream

typeInfo - The TypeInformation for the produced data stream.

Returns:

A data stream representing the elements in the iterator

readTextFile

public DataStreamSource<String> readTextFile(String filePath)

Creates a data stream that represents the Strings produced by reading the given file line wise. The file will be read with the system's default character set.

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path").

Returns:

The data stream that represents the data read from the given file as text lines

readTextFile

public DataStreamSource<String> readTextFile(String filePath,
                                             String charsetName)

Creates a data stream that represents the Strings produced by reading the given file line wise. The Charset with the given name will be used to read the files.

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path")

charsetName - The name of the character set used to read the file

Returns:

The data stream that represents the data read from the given file as text lines

readTextFileWithValue

public DataStreamSource<org.apache.flink.types.StringValue> readTextFileWithValue(String filePath)

Creates a data stream that represents the strings produced by reading the given file line wise. This method is similar to readTextFile(String), but it produces a data stream with mutable StringValue objects, rather than Java Strings. StringValues can be used to tune implementations to be less object and garbage collection heavy.

The file will be read with the system's default character set.

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path")

Returns:

A data stream that represents the data read from the given file as text lines

readTextFileWithValue

public DataStreamSource<org.apache.flink.types.StringValue> readTextFileWithValue(String filePath,
                                                                                  String charsetName,
                                                                                  boolean skipInvalidLines)

Creates a data stream that represents the Strings produced by reading the given file line wise. This method is similar to readTextFile(String, String), but it produces a data stream with mutable StringValue objects, rather than Java Strings. StringValues can be used to tune implementations to be less object and garbage collection heavy.

The Charset with the given name will be used to read the files.

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path")

charsetName - The name of the character set used to read the file

skipInvalidLines - A flag to indicate whether to skip lines that cannot be read with the given character set

Returns:

A data stream that represents the data read from the given file as text lines

readFile

public <OUT> DataStreamSource<OUT> readFile(org.apache.flink.api.common.io.FileInputFormat<OUT> inputFormat,
                                            String filePath)

Reads the given file with the given imput format.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path")

inputFormat - The input format used to create the data stream

Returns:

The data stream that represents the data read from the given file

readFileOfPrimitives

public <OUT> DataStreamSource<OUT> readFileOfPrimitives(String filePath,
                                                        Class<OUT> typeClass)

Creates a data stream that represents the primitive type produced by reading the given file line wise.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path")

typeClass - The primitive type class to be read

Returns:

A data stream that represents the data read from the given file as primitive type

readFileOfPrimitives

public <OUT> DataStreamSource<OUT> readFileOfPrimitives(String filePath,
                                                        String delimiter,
                                                        Class<OUT> typeClass)

Creates a data stream that represents the primitive type produced by reading the given file in delimited way.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path")

delimiter - The delimiter of the given file

typeClass - The primitive type class to be read

Returns:

A data stream that represents the data read from the given file as primitive type.

readFileStream

public DataStream<String> readFileStream(String filePath,
                                         long intervalMillis,
                                         FileMonitoringFunction.WatchType watchType)

Creates a data stream that contains the contents of file created while system watches the given path. The file will be read with the system's default character set.

Parameters:

filePath - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path/")

intervalMillis - The interval of file watching in milliseconds

watchType - The watch type of file stream. When watchType is FileMonitoringFunction.WatchType.ONLY_NEW_FILES, the system processes only new files. FileMonitoringFunction.WatchType.REPROCESS_WITH_APPENDED means that the system re-processes all contents of appended file. FileMonitoringFunction.WatchType.PROCESS_ONLY_APPENDED means that the system processes only appended contents of files.

Returns:

The DataStream containing the given directory.

socketTextStream

public DataStreamSource<String> socketTextStream(String hostname,
                                                 int port,
                                                 char delimiter,
                                                 long maxRetry)

Creates a new data stream that contains the strings received infinitely from a socket. Received strings are decoded by the system's default character set. On the termination of the socket server connection retries can be initiated.

Let us note that the socket itself does not report on abort and as a consequence retries are only initiated when the socket was gracefully terminated.

Parameters:

hostname - The host name which a server socket binds

port - The port number which a server socket binds. A port number of 0 means that the port number is automatically allocated.

delimiter - A character which splits received strings into records

maxRetry - The maximal retry interval in seconds while the program waits for a socket that is temporarily down. Reconnection is initiated every second. A number of 0 means that the reader is immediately terminated, while a negative value ensures retrying forever.

Returns:

A data stream containing the strings received from the socket

socketTextStream

public DataStreamSource<String> socketTextStream(String hostname,
                                                 int port,
                                                 char delimiter)

Creates a new data stream that contains the strings received infinitely from a socket. Received strings are decoded by the system's default character set. The reader is terminated immediately when the socket is down.

Parameters:

hostname - The host name which a server socket binds

port - The port number which a server socket binds. A port number of 0 means that the port number is automatically allocated.

delimiter - A character which splits received strings into records

Returns:

A data stream containing the strings received from the socket

socketTextStream

public DataStreamSource<String> socketTextStream(String hostname,
                                                 int port)

Creates a new data stream that contains the strings received infinitely from a socket. Received strings are decoded by the system's default character set, using'\n' as delimiter. The reader is terminated immediately when the socket is down.

Parameters:

hostname - The host name which a server socket binds

port - The port number which a server socket binds. A port number of 0 means that the port number is automatically allocated.

Returns:

A data stream containing the strings received from the socket

createInput

public <OUT> DataStreamSource<OUT> createInput(org.apache.flink.api.common.io.InputFormat<OUT,?> inputFormat)

Generic method to create an input data stream with InputFormat.

Since all data streams need specific information about their types, this method needs to determine the type of the data produced by the input format. It will attempt to determine the data type by reflection, unless the input format implements the ResultTypeQueryable interface. In the latter case, this method will invoke the ResultTypeQueryable.getProducedType() method to determine data type produced by the input format.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

inputFormat - The input format used to create the data stream

Returns:

The data stream that represents the data created by the input format

createInput

public <OUT> DataStreamSource<OUT> createInput(org.apache.flink.api.common.io.InputFormat<OUT,?> inputFormat,
                                               org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo)

Generic method to create an input data stream with InputFormat.

The data stream is typed to the given TypeInformation. This method is intended for input formats where the return type cannot be determined by reflection analysis, and that do not implement the ResultTypeQueryable interface.

Type Parameters:

OUT - The type of the returned data stream

Parameters:

inputFormat - The input format used to create the data stream

Returns:

The data stream that represents the data created by the input format

addSource

public <OUT> DataStreamSource<OUT> addSource(SourceFunction<OUT> function)

Adds a data source with a custom type information thus opening a DataStream. Only in very special cases does the user need to support type information. Otherwise use addSource(org.apache.flink.streaming.api.functions.source.SourceFunction)

By default sources have a parallelism of 1. To enable parallel execution, the user defined source should implement ParallelSourceFunction or extend RichParallelSourceFunction. In these cases the resulting source will have the parallelism of the environment. To change this afterwards call DataStreamSource.setParallelism(int)

Type Parameters:

OUT - type of the returned stream

Parameters:

function - the user defined function

Returns:

the data stream constructed

addSource

public <OUT> DataStreamSource<OUT> addSource(SourceFunction<OUT> function,
                                             String sourceName)

Ads a data source with a custom type information thus opening a DataStream. Only in very special cases does the user need to support type information. Otherwise use addSource(org.apache.flink.streaming.api.functions.source.SourceFunction)

Type Parameters:

OUT - type of the returned stream

Parameters:

function - the user defined function

sourceName - Name of the data source

Returns:

the data stream constructed

getExecutionEnvironment

public static StreamExecutionEnvironment getExecutionEnvironment()

Creates an execution environment that represents the context in which the program is currently executed. If the program is invoked standalone, this method returns a local execution environment, as returned by createLocalEnvironment().

Returns:

The execution environment of the context in which the program is executed.

createLocalEnvironment

public static LocalStreamEnvironment createLocalEnvironment()

Creates a LocalStreamEnvironment. The local execution environment will run the program in a multi-threaded fashion in the same JVM as the environment was created in. The default parallelism of the local environment is the number of hardware contexts (CPU cores / threads), unless it was specified differently by setParallelism(int).

Returns:

A local execution environment.

createLocalEnvironment

public static LocalStreamEnvironment createLocalEnvironment(int parallelism)

Creates a LocalStreamEnvironment. The local execution environment will run the program in a multi-threaded fashion in the same JVM as the environment was created in. It will use the parallelism specified in the parameter.

Parameters:

parallelism - The parallelism for the local environment.

Returns:

A local execution environment with the specified parallelism.

createRemoteEnvironment

public static StreamExecutionEnvironment createRemoteEnvironment(String host,
                                                                 int port,
                                                                 String... jarFiles)

Creates a RemoteStreamEnvironment. The remote environment sends (parts of) the program to a cluster for execution. Note that all file paths used in the program must be accessible from the cluster. The execution will use no parallelism, unless the parallelism is set explicitly via setParallelism(int).

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

jarFiles - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

Returns:

A remote environment that executes the program on a cluster.

createRemoteEnvironment

public static StreamExecutionEnvironment createRemoteEnvironment(String host,
                                                                 int port,
                                                                 int parallelism,
                                                                 String... jarFiles)

Creates a RemoteStreamEnvironment. The remote environment sends (parts of) the program to a cluster for execution. Note that all file paths used in the program must be accessible from the cluster. The execution will use the specified parallelism.

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

parallelism - The parallelism to use during the execution.

jarFiles - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

Returns:

A remote environment that executes the program on a cluster.

execute

public abstract org.apache.flink.api.common.JobExecutionResult execute()
                                                                throws Exception

Triggers the program execution. The environment will execute all parts of the program that have resulted in a "sink" operation. Sink operations are for example printing results or forwarding them to a message queue.

The program execution will be logged and displayed with a generated default name.

Returns:

The result of the job execution, containing elapsed time and accumulators.

Throws:

Exception

execute

public abstract org.apache.flink.api.common.JobExecutionResult execute(String jobName)
                                                                throws Exception

Triggers the program execution. The environment will execute all parts of the program that have resulted in a "sink" operation. Sink operations are for example printing results or forwarding them to a message queue.

The program execution will be logged and displayed with the provided name

Parameters:

jobName - Desired name of the job

Returns:

The result of the job execution, containing elapsed time and accumulators.

Throws:

Exception

getStreamGraph

public StreamGraph getStreamGraph()

Getter of the StreamGraph of the streaming job.

Returns:

The streamgraph representing the transformations

getExecutionPlan

public String getExecutionPlan()

Creates the plan with which the system will execute the program, and returns it as a String using a JSON representation of the execution data flow graph. Note that this needs to be called, before the plan is executed.

Returns:

The execution plan of the program, as a JSON String.

initializeFromFactory

protected static void initializeFromFactory(StreamExecutionEnvironmentFactory eef)

clean

public <F> F clean(F f)

Returns a "closure-cleaned" version of the given function. Cleans only if closure cleaning is not disabled in the ExecutionConfig