org.apache.hudi.source

Class IncrementalInputSplits

java.lang.Object

org.apache.hudi.source.IncrementalInputSplits

All Implemented Interfaces:

Serializable

public class IncrementalInputSplits
extends Object
implements scala.Serializable

Utilities to generate incremental input splits MergeOnReadInputSplit. The input splits are used for streaming and incremental read.

How to generate the input splits:

1. first fetch all the commit metadata for the incremental instants;
2. resolve the incremental commit file paths;
3. filter the full file paths by required partitions;
4. use the file paths from #step 3 as the back-up of the filesystem view.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IncrementalInputSplits.Builder Builder for IncrementalInputSplits.
static class	IncrementalInputSplits.Result Represents a result of calling inputSplits(org.apache.hudi.common.table.HoodieTableMetaClient, org.apache.hadoop.conf.Configuration).

Method Summary

Modifier and Type	Method and Description
static IncrementalInputSplits.Builder	builder() Returns the builder.
IncrementalInputSplits.Result	inputSplits(HoodieTableMetaClient metaClient, org.apache.hadoop.conf.Configuration hadoopConf) Returns the incremental input splits.
IncrementalInputSplits.Result	inputSplits(HoodieTableMetaClient metaClient, org.apache.hadoop.conf.Configuration hadoopConf, String issuedInstant) Returns the incremental input splits.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static IncrementalInputSplits.Builder builder()

Returns the builder.

inputSplits

public IncrementalInputSplits.Result inputSplits(HoodieTableMetaClient metaClient,
                                                 org.apache.hadoop.conf.Configuration hadoopConf)

Returns the incremental input splits.

Parameters:

metaClient - The meta client

hadoopConf - The hadoop configuration

Returns:

The list of incremental input splits or empty if there are no new instants

inputSplits

public IncrementalInputSplits.Result inputSplits(HoodieTableMetaClient metaClient,
                                                 org.apache.hadoop.conf.Configuration hadoopConf,
                                                 String issuedInstant)

Returns the incremental input splits.

Parameters:

metaClient - The meta client

hadoopConf - The hadoop configuration

issuedInstant - The last issued instant, only valid in streaming read

Returns:

The list of incremental input splits or empty if there are no new instants