org.apache.flink.streaming.connectors.elasticsearch2

Class ElasticsearchSink<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>

org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase<T>

org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink<T>

Type Parameters:

T - Type of the elements handled by this sink

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, CheckpointedFunction, SinkFunction<T>

public class ElasticsearchSink<T>
extends ElasticsearchSinkBase<T>

Elasticsearch 2.x sink that requests multiple ActionRequests against a cluster for each incoming element.

The sink internally uses a TransportClient to communicate with an Elasticsearch cluster. The sink will fail if no cluster can be connected to using the provided transport addresses passed to the constructor.

The Map passed to the constructor is used to create the TransportClient. The config keys can be found in the Elasticsearch documentation. An important setting is cluster.name, which should be set to the name of the cluster that the sink should emit to.

Internally, the sink will use a BulkProcessor to send ActionRequests. This will buffer elements before sending a request to the cluster. The behaviour of the BulkProcessor can be configured using these config keys:

• bulk.flush.max.actions: Maximum amount of elements to buffer
• bulk.flush.max.size.mb: Maximum amount of data (in megabytes) to buffer
• bulk.flush.interval.ms: Interval at which to flush data regardless of the other two settings in milliseconds

You also have to provide an ElasticsearchSinkFunction. This is used to create multiple ActionRequests for each incoming element. See the class level documentation of ElasticsearchSinkFunction for an example.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase

ElasticsearchSinkBase.BulkFlushBackoffPolicy, ElasticsearchSinkBase.FlushBackoffType

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase

CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY, CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE, CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES, CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE, CONFIG_KEY_BULK_FLUSH_INTERVAL_MS, CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS, CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

Constructor Summary

Constructors

Constructor and Description
ElasticsearchSink(Map<String,String> userConfig, List<InetSocketAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction) Deprecated. Deprecated since 1.2, to be removed at 2.0. Please use ElasticsearchSink(Map, List, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction) instead.
ElasticsearchSink(Map<String,String> userConfig, List<InetSocketAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction) Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.
ElasticsearchSink(Map<String,String> userConfig, List<InetSocketAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction, ActionRequestFailureHandler failureHandler) Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Method Summary

Methods inherited from class org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase

buildBulkProcessor, close, disableFlushOnCheckpoint, initializeState, invoke, open, snapshotState

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ElasticsearchSink

@Deprecated
public ElasticsearchSink(Map<String,String> userConfig,
                            List<InetSocketAddress> transportAddresses,
                            ElasticsearchSinkFunction<T> elasticsearchSinkFunction)

Deprecated. Deprecated since 1.2, to be removed at 2.0. Please use ElasticsearchSink(Map, List, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction) instead.

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessor

transportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClient

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                 List<InetSocketAddress> transportAddresses,
                 ElasticsearchSinkFunction<T> elasticsearchSinkFunction)

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are passed when constructing the TransportClient and BulkProcessor

transportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClient

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                 List<InetSocketAddress> transportAddresses,
                 ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
                 ActionRequestFailureHandler failureHandler)

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessor

transportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClient

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

failureHandler - This is used to handle failed ActionRequest