org.apache.beam.sdk.io.elasticsearch

Class ElasticsearchIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<java.lang.String>,org.apache.beam.sdk.values.PCollectionTuple>

org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

ElasticsearchIO

public static class ElasticsearchIO.Write
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<java.lang.String>,org.apache.beam.sdk.values.PCollectionTuple>

A PTransform writing data to Elasticsearch.

This PTransform acts as a convenience wrapper for doing both document to bulk API serialization as well as batching those Bulk API entities and writing them to an Elasticsearch cluster. This class is effectively a thin proxy for DocToBulk->BulkIO all-in-one for convenience and backward compatibility.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ElasticsearchIO.Write.BooleanFieldValueExtractFn
static interface	ElasticsearchIO.Write.FieldValueExtractFn

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.beam.sdk.values.TupleTag<ElasticsearchIO.Document>	FAILED_WRITES
static org.apache.beam.sdk.values.TupleTag<ElasticsearchIO.Document>	SUCCESSFUL_WRITES

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.values.PCollectionTuple	expand(org.apache.beam.sdk.values.PCollection<java.lang.String> input)
ElasticsearchIO.BulkIO	getBulkIO()
ElasticsearchIO.DocToBulk	getDocToBulk()
ElasticsearchIO.Write	withAllowableResponseErrors(@Nullable java.util.Set<java.lang.String> allowableResponseErrors) Refer to ElasticsearchIO.BulkIO.withAllowableResponseErrors(java.util.Set<java.lang.String>).
ElasticsearchIO.Write	withAppendOnly(boolean appendOnly) Refer to ElasticsearchIO.DocToBulk.withAppendOnly(boolean).
ElasticsearchIO.Write	withBackendVersion(int backendVersion) Refer to ElasticsearchIO.DocToBulk.withBackendVersion(int).
ElasticsearchIO.Write	withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration) Refer to ElasticsearchIO.BulkIO.withConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration).
ElasticsearchIO.Write	withDocVersionFn(ElasticsearchIO.Write.FieldValueExtractFn docVersionFn) Refer to ElasticsearchIO.DocToBulk.withDocVersionFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withDocVersionType(java.lang.String docVersionType) Refer to ElasticsearchIO.DocToBulk.withDocVersionType(java.lang.String).
ElasticsearchIO.Write	withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn) Refer to ElasticsearchIO.DocToBulk.withIdFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withIgnoreVersionConflicts(boolean ignoreVersionConflicts) Refer to ElasticsearchIO.BulkIO.withIgnoreVersionConflicts(boolean).
ElasticsearchIO.Write	withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn) Refer to ElasticsearchIO.DocToBulk.withIndexFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn) Refer to ElasticsearchIO.DocToBulk.withIsDeleteFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.BooleanFieldValueExtractFn).
ElasticsearchIO.Write	withMaxBatchSize(long batchSize) Refer to ElasticsearchIO.BulkIO.withMaxBatchSize(long).
ElasticsearchIO.Write	withMaxBatchSizeBytes(long batchSizeBytes) Refer to ElasticsearchIO.BulkIO.withMaxBatchSizeBytes(long).
ElasticsearchIO.Write	withMaxBufferingDuration(org.joda.time.Duration maxBufferingDuration) Refer to ElasticsearchIO.BulkIO.withMaxBufferingDuration(org.joda.time.Duration).
ElasticsearchIO.Write	withMaxParallelRequestsPerWindow(int maxParallelRequestsPerWindow) Refer to ElasticsearchIO.BulkIO.withMaxParallelRequestsPerWindow(int).
ElasticsearchIO.Write	withRetryConfiguration(ElasticsearchIO.RetryConfiguration retryConfiguration) Refer to ElasticsearchIO.BulkIO.withRetryConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration).
ElasticsearchIO.Write	withRoutingFn(ElasticsearchIO.Write.FieldValueExtractFn routingFn) Refer to ElasticsearchIO.DocToBulk.withRoutingFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withThrowWriteErrors(boolean throwWriteErrors) Refer to ElasticsearchIO.BulkIO.withThrowWriteErrors(boolean).
ElasticsearchIO.Write	withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn) Refer to ElasticsearchIO.DocToBulk.withTypeFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withUpsertScript(java.lang.String source) Refer to ElasticsearchIO.DocToBulk.withUpsertScript(java.lang.String).
ElasticsearchIO.Write	withUsePartialUpdate(boolean usePartialUpdate) Refer to ElasticsearchIO.DocToBulk.withUsePartialUpdate(boolean).
ElasticsearchIO.Write	withUseStatefulBatches(boolean useStatefulBatches) Refer to ElasticsearchIO.BulkIO.withUseStatefulBatches(boolean).

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SUCCESSFUL_WRITES

public static final org.apache.beam.sdk.values.TupleTag<ElasticsearchIO.Document> SUCCESSFUL_WRITES

FAILED_WRITES

public static final org.apache.beam.sdk.values.TupleTag<ElasticsearchIO.Document> FAILED_WRITES

Constructor Detail

Write

public Write()

Method Detail

getDocToBulk

public ElasticsearchIO.DocToBulk getDocToBulk()

getBulkIO

public ElasticsearchIO.BulkIO getBulkIO()

withIdFn

public ElasticsearchIO.Write withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)

Refer to ElasticsearchIO.DocToBulk.withIdFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withIndexFn

public ElasticsearchIO.Write withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)

Refer to ElasticsearchIO.DocToBulk.withIndexFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withRoutingFn

public ElasticsearchIO.Write withRoutingFn(ElasticsearchIO.Write.FieldValueExtractFn routingFn)

Refer to ElasticsearchIO.DocToBulk.withRoutingFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withTypeFn

public ElasticsearchIO.Write withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)

Refer to ElasticsearchIO.DocToBulk.withTypeFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withDocVersionFn

public ElasticsearchIO.Write withDocVersionFn(ElasticsearchIO.Write.FieldValueExtractFn docVersionFn)

Refer to ElasticsearchIO.DocToBulk.withDocVersionFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withDocVersionType

public ElasticsearchIO.Write withDocVersionType(java.lang.String docVersionType)

Refer to ElasticsearchIO.DocToBulk.withDocVersionType(java.lang.String).

withUsePartialUpdate

public ElasticsearchIO.Write withUsePartialUpdate(boolean usePartialUpdate)

Refer to ElasticsearchIO.DocToBulk.withUsePartialUpdate(boolean).

withAppendOnly

public ElasticsearchIO.Write withAppendOnly(boolean appendOnly)

Refer to ElasticsearchIO.DocToBulk.withAppendOnly(boolean).

withUpsertScript

public ElasticsearchIO.Write withUpsertScript(java.lang.String source)

Refer to ElasticsearchIO.DocToBulk.withUpsertScript(java.lang.String).

withBackendVersion

public ElasticsearchIO.Write withBackendVersion(int backendVersion)

Refer to ElasticsearchIO.DocToBulk.withBackendVersion(int).

withIsDeleteFn

public ElasticsearchIO.Write withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn)

Refer to ElasticsearchIO.DocToBulk.withIsDeleteFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.BooleanFieldValueExtractFn).

withConnectionConfiguration

public ElasticsearchIO.Write withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)

Refer to ElasticsearchIO.BulkIO.withConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration).

withMaxBatchSize

public ElasticsearchIO.Write withMaxBatchSize(long batchSize)

Refer to ElasticsearchIO.BulkIO.withMaxBatchSize(long).

withMaxBatchSizeBytes

public ElasticsearchIO.Write withMaxBatchSizeBytes(long batchSizeBytes)

Refer to ElasticsearchIO.BulkIO.withMaxBatchSizeBytes(long).

withRetryConfiguration

public ElasticsearchIO.Write withRetryConfiguration(ElasticsearchIO.RetryConfiguration retryConfiguration)

Refer to ElasticsearchIO.BulkIO.withRetryConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration).

withIgnoreVersionConflicts

public ElasticsearchIO.Write withIgnoreVersionConflicts(boolean ignoreVersionConflicts)

Refer to ElasticsearchIO.BulkIO.withIgnoreVersionConflicts(boolean).

withUseStatefulBatches

public ElasticsearchIO.Write withUseStatefulBatches(boolean useStatefulBatches)

Refer to ElasticsearchIO.BulkIO.withUseStatefulBatches(boolean).

withMaxBufferingDuration

public ElasticsearchIO.Write withMaxBufferingDuration(org.joda.time.Duration maxBufferingDuration)

Refer to ElasticsearchIO.BulkIO.withMaxBufferingDuration(org.joda.time.Duration).

withMaxParallelRequestsPerWindow

public ElasticsearchIO.Write withMaxParallelRequestsPerWindow(int maxParallelRequestsPerWindow)

Refer to ElasticsearchIO.BulkIO.withMaxParallelRequestsPerWindow(int).

withAllowableResponseErrors

public ElasticsearchIO.Write withAllowableResponseErrors(@Nullable java.util.Set<java.lang.String> allowableResponseErrors)

Refer to ElasticsearchIO.BulkIO.withAllowableResponseErrors(java.util.Set<java.lang.String>).

withThrowWriteErrors

public ElasticsearchIO.Write withThrowWriteErrors(boolean throwWriteErrors)

Refer to ElasticsearchIO.BulkIO.withThrowWriteErrors(boolean).

expand

public org.apache.beam.sdk.values.PCollectionTuple expand(org.apache.beam.sdk.values.PCollection<java.lang.String> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<java.lang.String>,org.apache.beam.sdk.values.PCollectionTuple>