Package org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Read

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    Enclosing class:
    KinesisIO
    public abstract static class KinesisIO.Read
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<KinesisRecord>>
    Implementation of KinesisIO.read().
    See Also:
    Serialized Form

    • Constructor Detail

      • Read

        public Read()

    • Method Detail

      • withStreamName

        public KinesisIO.Read withStreamName​(java.lang.String streamName)
        Specify reading from streamName.

      • withInitialPositionInStream

        public KinesisIO.Read withInitialPositionInStream​(software.amazon.kinesis.common.InitialPositionInStream initialPosition)
        Specify reading from some initial position in stream.

      • withInitialTimestampInStream

        public KinesisIO.Read withInitialTimestampInStream​(org.joda.time.Instant initialTimestamp)
        Specify reading beginning at given Instant. This Instant must be in the past, i.e. before Instant.now().

      • withAWSClientsProvider

        @Deprecated
public KinesisIO.Read withAWSClientsProvider​(java.lang.String awsAccessKey,
                                             java.lang.String awsSecretKey,
                                             software.amazon.awssdk.regions.Region region,
                                             java.lang.String serviceEndpoint)
        Deprecated.
        Use withClientConfiguration(ClientConfiguration) instead.

      • withAWSClientsProvider

        @Deprecated
public KinesisIO.Read withAWSClientsProvider​(software.amazon.awssdk.auth.credentials.AwsCredentialsProvider awsCredentialsProvider,
                                             software.amazon.awssdk.regions.Region region)
        Deprecated.
        Use withClientConfiguration(ClientConfiguration) instead.

      • withAWSClientsProvider

        @Deprecated
public KinesisIO.Read withAWSClientsProvider​(software.amazon.awssdk.auth.credentials.AwsCredentialsProvider awsCredentialsProvider,
                                             software.amazon.awssdk.regions.Region region,
                                             java.lang.String serviceEndpoint)
        Deprecated.
        Use withClientConfiguration(ClientConfiguration) instead.

      • withMaxNumRecords

        public KinesisIO.Read withMaxNumRecords​(long maxNumRecords)
        Specifies to read at most a given number of records.

      • withMaxReadTime

        public KinesisIO.Read withMaxReadTime​(org.joda.time.Duration maxReadTime)
        Specifies to read records during maxReadTime.

      • withUpToDateThreshold

        public KinesisIO.Read withUpToDateThreshold​(org.joda.time.Duration upToDateThreshold)
        Specifies how late records consumed by this source can be to still be considered on time. When this limit is exceeded the actual backlog size will be evaluated and the runner might decide to scale the amount of resources allocated to the pipeline in order to speed up ingestion.

      • withRequestRecordsLimit

        public KinesisIO.Read withRequestRecordsLimit​(int limit)
        Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records. If should be adjusted according to average size of data record to prevent shard overloading. More details can be found here: API_GetRecords

      • withArrivalTimeWatermarkPolicy

        public KinesisIO.Read withArrivalTimeWatermarkPolicy()
        Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

      • withArrivalTimeWatermarkPolicy

        public KinesisIO.Read withArrivalTimeWatermarkPolicy​(org.joda.time.Duration watermarkIdleDurationThreshold)
        Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

        {@param watermarkIdleDurationThreshold} Denotes the duration for which the watermark can be idle.

      • withProcessingTimeWatermarkPolicy

        public KinesisIO.Read withProcessingTimeWatermarkPolicy()
        Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.

      • withCustomWatermarkPolicy

        public KinesisIO.Read withCustomWatermarkPolicy​(WatermarkPolicyFactory watermarkPolicyFactory)
        Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.
        Parameters:
        watermarkPolicyFactory - Custom Watermark policy factory.

      • withFixedDelayRateLimitPolicy

        public KinesisIO.Read withFixedDelayRateLimitPolicy()
        Specifies a fixed delay rate limit policy with the default delay of 1 second.

      • withFixedDelayRateLimitPolicy

        public KinesisIO.Read withFixedDelayRateLimitPolicy​(org.joda.time.Duration delay)
        Specifies a fixed delay rate limit policy with the given delay.
        Parameters:
        delay - Denotes the fixed delay duration.

      • withDynamicDelayRateLimitPolicy

        public KinesisIO.Read withDynamicDelayRateLimitPolicy​(java.util.function.Supplier<org.joda.time.Duration> delay)
        Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value. This can be used to change the polling interval of a running pipeline based on some external configuration source, for example.
        Parameters:
        delay - The function to invoke to get the next delay duration.

      • withCustomRateLimitPolicy

        public KinesisIO.Read withCustomRateLimitPolicy​(RateLimitPolicyFactory rateLimitPolicyFactory)
        Specifies the RateLimitPolicyFactory for a custom rate limiter.
        Parameters:
        rateLimitPolicyFactory - Custom rate limit policy factory.

      • withMaxCapacityPerShard

        public KinesisIO.Read withMaxCapacityPerShard​(java.lang.Integer maxCapacity)
        Specifies the maximum number of messages per one shard.

      • expand

        public org.apache.beam.sdk.values.PCollection<KinesisRecord> expand​(org.apache.beam.sdk.values.PBegin input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<KinesisRecord>>