org.apache.flink.streaming.connectors.elasticsearch

Class ElasticsearchSinkBase<T,C extends AutoCloseable>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>

org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase<T,C>

Type Parameters:

T - Type of the elements handled by this sink

C - Type of the Elasticsearch client, which implements AutoCloseable

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>

@Internal
public abstract class ElasticsearchSinkBase<T,C extends AutoCloseable>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Base class for all Flink Elasticsearch Sinks.

This class implements the common behaviour across Elasticsearch versions, such as the use of an internal BulkProcessor to buffer multiple ActionRequests before sending the requests to the cluster, as well as passing input records to the user provided ElasticsearchSinkFunction for processing.

The version specific API calls for different Elasticsearch versions should be defined by a concrete implementation of a ElasticsearchApiCallBridge, which is provided to the constructor of this class. This call bridge is used, for example, to create a Elasticsearch Client, handle failed item responses, etc.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ElasticsearchSinkBase.BulkFlushBackoffPolicy Provides a backoff policy for bulk requests.
static class	ElasticsearchSinkBase.FlushBackoffType Used to control whether the retry delay should increase exponentially or remain constant.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY
static String	CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE
static String	CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES
static String	CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE
static String	CONFIG_KEY_BULK_FLUSH_INTERVAL_MS
static String	CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS
static String	CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

Constructor Summary

Constructors

Constructor and Description
ElasticsearchSinkBase(ElasticsearchApiCallBridge<C> callBridge, Map<String,String> userConfig, ElasticsearchSinkFunction<T> elasticsearchSinkFunction, ActionRequestFailureHandler failureHandler)

Method Summary

Modifier and Type	Method and Description
protected org.elasticsearch.action.bulk.BulkProcessor	buildBulkProcessor(org.elasticsearch.action.bulk.BulkProcessor.Listener listener) Build the BulkProcessor.
void	close()
void	disableFlushOnCheckpoint() Disable flushing on checkpoint.
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	invoke(T value, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
void	open(org.apache.flink.configuration.Configuration parameters)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

finish, invoke, writeWatermark

Field Detail

CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS

public static final String CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

public static final String CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_INTERVAL_MS

public static final String CONFIG_KEY_BULK_FLUSH_INTERVAL_MS

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE

public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE

public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES

public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES

See Also:

Constant Field Values

CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY

public static final String CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY

See Also:

Constant Field Values

Constructor Detail

ElasticsearchSinkBase

public ElasticsearchSinkBase(ElasticsearchApiCallBridge<C> callBridge,
                             Map<String,String> userConfig,
                             ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
                             ActionRequestFailureHandler failureHandler)

Method Detail

disableFlushOnCheckpoint

public void disableFlushOnCheckpoint()

Disable flushing on checkpoint. When disabled, the sink will not wait for all pending action requests to be acknowledged by Elasticsearch on checkpoints.

NOTE: If flushing on checkpoint is disabled, the Flink Elasticsearch Sink does NOT provide any strong guarantees for at-least-once delivery of action requests.

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

invoke

public void invoke(T value,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws Exception

Specified by:

invoke in interface org.apache.flink.streaming.api.functions.sink.SinkFunction<T>

Throws:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

buildBulkProcessor

@VisibleForTesting
protected org.elasticsearch.action.bulk.BulkProcessor buildBulkProcessor(org.elasticsearch.action.bulk.BulkProcessor.Listener listener)

Build the BulkProcessor.

Note: this is exposed for testing purposes.