org.apache.flink.streaming.connectors.elasticsearch

Class ElasticsearchSink<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>

org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase<T>

org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSink<T>

Type Parameters:

T - Type of the elements handled by this sink

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>

public class ElasticsearchSink<T>
extends org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase<T>

Elasticsearch 1.x sink that requests multiple ActionRequests against a cluster for each incoming element.

When using the first constructor ElasticsearchSink(java.util.Map, ElasticsearchSinkFunction) the sink will create a local Node for communicating with the Elasticsearch cluster. When using the second constructor ElasticsearchSink(java.util.Map, java.util.List, ElasticsearchSinkFunction) a TransportClient will be used instead.

Attention: When using the TransportClient the sink will fail if no cluster can be connected to. When using the local Node for communicating, the sink will block and wait for a cluster to come online.

The Map passed to the constructor is used to create the Node or TransportClient. The config keys can be found in the Elasticsearch documentation. An important setting is cluster.name, which should be set to the name of the cluster that the sink should emit to.

Internally, the sink will use a BulkProcessor to send ActionRequests. This will buffer elements before sending a request to the cluster. The behaviour of the BulkProcessor can be configured using these config keys:

• bulk.flush.max.actions: Maximum amount of elements to buffer
• bulk.flush.max.size.mb: Maximum amount of data (in megabytes) to buffer
• bulk.flush.interval.ms: Interval at which to flush data regardless of the other two settings in milliseconds

You also have to provide an ElasticsearchSinkFunction. This is used to create multiple ActionRequests for each incoming element. See the class level documentation of ElasticsearchSinkFunction for an example.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase

org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.BulkFlushBackoffPolicy, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.FlushBackoffType

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase

CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY, CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE, CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES, CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE, CONFIG_KEY_BULK_FLUSH_INTERVAL_MS, CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS, CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

Constructor Summary

Constructors

Constructor and Description
ElasticsearchSink(Map<String,String> userConfig, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction) Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.
ElasticsearchSink(Map<String,String> userConfig, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction, org.apache.flink.streaming.connectors.elasticsearch.ActionRequestFailureHandler failureHandler) Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.
ElasticsearchSink(Map<String,String> userConfig, IndexRequestBuilder<T> indexRequestBuilder) Deprecated. Deprecated since version 1.2, to be removed at version 2.0. Please use ElasticsearchSink(Map, ElasticsearchSinkFunction) instead.
ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportAddresses, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction) Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.
ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportAddresses, org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction, org.apache.flink.streaming.connectors.elasticsearch.ActionRequestFailureHandler failureHandler) Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.
ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportAddresses, IndexRequestBuilder<T> indexRequestBuilder) Deprecated. Deprecated since 1.2, to be removed at 2.0. Please use ElasticsearchSink(Map, List, ElasticsearchSinkFunction) instead.

Method Summary

Methods inherited from class org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase

buildBulkProcessor, close, disableFlushOnCheckpoint, initializeState, invoke, open, snapshotState

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ElasticsearchSink

@Deprecated
public ElasticsearchSink(Map<String,String> userConfig,
                                     IndexRequestBuilder<T> indexRequestBuilder)

Deprecated. Deprecated since version 1.2, to be removed at version 2.0. Please use ElasticsearchSink(Map, ElasticsearchSinkFunction) instead.

Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.

Parameters:

userConfig - The map of user settings that are used when constructing the Node and BulkProcessor

indexRequestBuilder - This is used to generate the IndexRequest from the incoming element

ElasticsearchSink

@Deprecated
public ElasticsearchSink(Map<String,String> userConfig,
                                     List<org.elasticsearch.common.transport.TransportAddress> transportAddresses,
                                     IndexRequestBuilder<T> indexRequestBuilder)

Deprecated. Deprecated since 1.2, to be removed at 2.0. Please use ElasticsearchSink(Map, List, ElasticsearchSinkFunction) instead.

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessor

transportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClient

indexRequestBuilder - This is used to generate a IndexRequest from the incoming element

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                         org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction)

Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.

Parameters:

userConfig - The map of user settings that are used when constructing the embedded Node and BulkProcessor

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                         List<org.elasticsearch.common.transport.TransportAddress> transportAddresses,
                         org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction)

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessor

transportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClient

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                         org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
                         org.apache.flink.streaming.connectors.elasticsearch.ActionRequestFailureHandler failureHandler)

Creates a new ElasticsearchSink that connects to the cluster using an embedded Node.

Parameters:

userConfig - The map of user settings that are used when constructing the embedded Node and BulkProcessor

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

failureHandler - This is used to handle failed ActionRequest

ElasticsearchSink

public ElasticsearchSink(Map<String,String> userConfig,
                         List<org.elasticsearch.common.transport.TransportAddress> transportAddresses,
                         org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
                         org.apache.flink.streaming.connectors.elasticsearch.ActionRequestFailureHandler failureHandler)

Creates a new ElasticsearchSink that connects to the cluster using a TransportClient.

Parameters:

userConfig - The map of user settings that are used when constructing the TransportClient and BulkProcessor

transportAddresses - The addresses of Elasticsearch nodes to which to connect using a TransportClient

elasticsearchSinkFunction - This is used to generate multiple ActionRequest from the incoming element

failureHandler - This is used to handle failed ActionRequest