org.apache.flink.streaming.python.api.environment

Class PythonStreamExecutionEnvironment

java.lang.Object

org.apache.flink.streaming.python.api.environment.PythonStreamExecutionEnvironment

@PublicEvolving
public class PythonStreamExecutionEnvironment
extends Object

A thin wrapper layer over StreamExecutionEnvironment.

The PythonStreamExecutionEnvironment is the context in which a streaming program is executed.

The environment provides methods to control the job execution (such as setting the parallelism or the fault tolerance/checkpointing parameters) and to interact with the outside world (data access).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PythonStreamExecutionEnvironment.PythonJobParameters Utility class for storing/retrieving parameters in/from a ExecutionConfig.GlobalJobParameters.

Method Summary

Modifier and Type	Method and Description
PythonDataStream	add_java_source(org.apache.flink.streaming.api.functions.source.SourceFunction<Object> src) Add a java source to the streaming topology.
PythonDataStream	create_python_source(org.apache.flink.streaming.api.functions.source.SourceFunction<Object> src)
PythonStreamExecutionEnvironment	enable_checkpointing(long interval) A thin wrapper layer over StreamExecutionEnvironment.enableCheckpointing(long).
PythonStreamExecutionEnvironment	enable_checkpointing(long interval, org.apache.flink.streaming.api.CheckpointingMode mode) A thin wrapper layer over StreamExecutionEnvironment.enableCheckpointing(long, CheckpointingMode).
org.apache.flink.api.common.JobExecutionResult	execute() A thin wrapper layer over StreamExecutionEnvironment.execute().
org.apache.flink.api.common.JobExecutionResult	execute(String job_name) A thin wrapper layer over StreamExecutionEnvironment.execute(java.lang.String).
PythonDataStream	from_collection(Collection<Object> collection) A thin wrapper layer over StreamExecutionEnvironment.fromCollection(java.util.Collection)
PythonDataStream	from_collection(Iterator<Object> iter) Creates a python data stream from the given iterator.
PythonDataStream	from_elements(org.python.core.PyObject... elements) A thin wrapper layer over StreamExecutionEnvironment.fromElements(java.lang.Object[]).
PythonDataStream	generate_sequence(long from, long to) A thin wrapper layer over StreamExecutionEnvironment.generateSequence(long, long).
PythonDataStream	read_text_file(String path) A thin wrapper layer over StreamExecutionEnvironment.readTextFile(java.lang.String).
PythonStreamExecutionEnvironment	set_parallelism(int parallelism) A thin wrapper layer over StreamExecutionEnvironment.setParallelism(int).
PythonDataStream	socket_text_stream(String host, int port) A thin wrapper layer over StreamExecutionEnvironment.socketTextStream(java.lang.String, int).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create_python_source

public PythonDataStream create_python_source(org.apache.flink.streaming.api.functions.source.SourceFunction<Object> src)
                                      throws Exception

Throws:

Exception

add_java_source

public PythonDataStream add_java_source(org.apache.flink.streaming.api.functions.source.SourceFunction<Object> src)

Add a java source to the streaming topology. The source expected to be an java based implementation (.e.g. Kafka connector).

Parameters:

src - A native java source (e.g. PythonFlinkKafkaConsumer09)

Returns:

Python data stream

from_elements

public PythonDataStream from_elements(org.python.core.PyObject... elements)

A thin wrapper layer over StreamExecutionEnvironment.fromElements(java.lang.Object[]).

Parameters:

elements - The array of PyObject elements to create the data stream from.

Returns:

The data stream representing the given array of elements

from_collection

public PythonDataStream from_collection(Collection<Object> collection)

A thin wrapper layer over StreamExecutionEnvironment.fromCollection(java.util.Collection)

The input Collection is of type Object, because it is a collection of Python elements. * There type is determined in runtime, by the Jython framework.

Parameters:

collection - The collection of python elements to create the data stream from.

Returns:

The data stream representing the given collection

from_collection

public PythonDataStream from_collection(Iterator<Object> iter)
                                 throws Exception

Creates a python data stream from the given iterator.

Note that this operation will result in a non-parallel data stream source, i.e., a data stream source with a parallelism of one.

Parameters:

iter - The iterator of elements to create the data stream from

Returns:

The data stream representing the elements in the iterator

Throws:

Exception

See Also:

StreamExecutionEnvironment.fromCollection(java.util.Iterator, org.apache.flink.api.common.typeinfo.TypeInformation)

generate_sequence

public PythonDataStream generate_sequence(long from,
                                          long to)

A thin wrapper layer over StreamExecutionEnvironment.generateSequence(long, long).

Parameters:

from - The number to start at (inclusive)

to - The number to stop at (inclusive)

Returns:

A python data stream, containing all number in the [from, to] interval

read_text_file

public PythonDataStream read_text_file(String path)
                                throws IOException

A thin wrapper layer over StreamExecutionEnvironment.readTextFile(java.lang.String).

Parameters:

path - The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path").

Returns:

The data stream that represents the data read from the given file as text lines

Throws:

IOException

socket_text_stream

public PythonDataStream socket_text_stream(String host,
                                           int port)

A thin wrapper layer over StreamExecutionEnvironment.socketTextStream(java.lang.String, int).

Parameters:

host - The host name which a server socket binds

port - The port number which a server socket binds. A port number of 0 means that the port number is automatically allocated.

Returns:

A python data stream containing the strings received from the socket

enable_checkpointing

public PythonStreamExecutionEnvironment enable_checkpointing(long interval)

A thin wrapper layer over StreamExecutionEnvironment.enableCheckpointing(long).

Parameters:

interval - Time interval between state checkpoints in milliseconds.

Returns:

The same PythonStreamExecutionEnvironment instance of the caller

enable_checkpointing

public PythonStreamExecutionEnvironment enable_checkpointing(long interval,
                                                             org.apache.flink.streaming.api.CheckpointingMode mode)

A thin wrapper layer over StreamExecutionEnvironment.enableCheckpointing(long, CheckpointingMode).

Parameters:

interval - Time interval between state checkpoints in milliseconds.

mode - The checkpointing mode, selecting between "exactly once" and "at least once" guaranteed.

Returns:

The same PythonStreamExecutionEnvironment instance of the caller

set_parallelism

public PythonStreamExecutionEnvironment set_parallelism(int parallelism)

A thin wrapper layer over StreamExecutionEnvironment.setParallelism(int).

Parameters:

parallelism - The parallelism

Returns:

The same PythonStreamExecutionEnvironment instance of the caller

execute

public org.apache.flink.api.common.JobExecutionResult execute()
                                                       throws Exception

A thin wrapper layer over StreamExecutionEnvironment.execute().

Returns:

The result of the job execution

Throws:

Exception

execute

public org.apache.flink.api.common.JobExecutionResult execute(String job_name)
                                                       throws Exception

A thin wrapper layer over StreamExecutionEnvironment.execute(java.lang.String).

Returns:

The result of the job execution, containing elapsed time and accumulators.

Throws:

Exception - which occurs during job execution.