org.apache.hudi.source

Class IncrementalInputSplits

java.lang.Object

org.apache.hudi.source.IncrementalInputSplits

All Implemented Interfaces:

Serializable

public class IncrementalInputSplits
extends Object
implements Serializable

Utilities to generate incremental input splits MergeOnReadInputSplit. The input splits are used for streaming and incremental read.

How to generate the input splits:

1. first fetch all the commit metadata for the incremental instants;
2. resolve the incremental commit file paths;
3. filter the full file paths by required partitions;
4. use the file paths from #step 3 as the back-up of the filesystem view.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IncrementalInputSplits.Builder Builder for IncrementalInputSplits.
static class	IncrementalInputSplits.Result Represents a result of calling inputSplits(org.apache.hudi.common.table.HoodieTableMetaClient, boolean).

Method Summary

Modifier and Type	Method and Description
static IncrementalInputSplits.Builder	builder() Returns the builder.
HoodieTimeline	filterInstantsAsPerUserConfigs(HoodieTimeline timeline) Filters out the unnecessary instants as per user specified configs.
List<HoodieInstant>	filterInstantsWithRange(HoodieTimeline commitTimeline, String issuedInstant) Returns the instants with a given issuedInstant to start from.
IncrementalInputSplits.Result	inputSplits(HoodieTableMetaClient metaClient, boolean cdcEnabled) Returns the incremental input splits.
IncrementalInputSplits.Result	inputSplits(HoodieTableMetaClient metaClient, String issuedInstant, String issuedOffset, boolean cdcEnabled) Returns the incremental input splits.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static IncrementalInputSplits.Builder builder()

Returns the builder.

inputSplits

public IncrementalInputSplits.Result inputSplits(HoodieTableMetaClient metaClient,
                                                 boolean cdcEnabled)

Returns the incremental input splits.

Parameters:

metaClient - The meta client

cdcEnabled - Whether cdc is enabled

Returns:

The list of incremental input splits or empty if there are no new instants

inputSplits

public IncrementalInputSplits.Result inputSplits(HoodieTableMetaClient metaClient,
                                                 @Nullable
                                                 String issuedInstant,
                                                 @Nullable
                                                 String issuedOffset,
                                                 boolean cdcEnabled)

Returns the incremental input splits.

Parameters:

metaClient - The meta client

issuedInstant - The last issued instant, only valid in streaming read

issuedOffset - The last issued offset, only valid in streaming read

cdcEnabled - Whether cdc is enabled

Returns:

The list of incremental input splits or empty if there are no new instants

filterInstantsWithRange

@VisibleForTesting
public List<HoodieInstant> filterInstantsWithRange(HoodieTimeline commitTimeline,
                                                                      @Nullable
                                                                      String issuedInstant)

Returns the instants with a given issuedInstant to start from.

Parameters:

commitTimeline - The completed commits timeline

issuedInstant - The last issued instant that has already been delivered to downstream

Returns:

the filtered hoodie instants

filterInstantsAsPerUserConfigs

@VisibleForTesting
public HoodieTimeline filterInstantsAsPerUserConfigs(HoodieTimeline timeline)

Filters out the unnecessary instants as per user specified configs.

Parameters:

timeline - The timeline

Returns:

the filtered timeline