org.apache.hudi.common.table.read

Class IncrementalQueryAnalyzer

java.lang.Object

org.apache.hudi.common.table.read.IncrementalQueryAnalyzer

public class IncrementalQueryAnalyzer
extends Object

Analyzer for incremental queries on the timeline, to filter instants based on specified ranges.

The analyzer is supplied the following information:

• The archived instants;
• The active instants;
• The instant filtering predicate, e.g the instant range with a "startTime" and "endTime"
• Whether the query starts from the "earliest" available instant;
• Whether the query ends to the "latest" available instant;
• The max completion time used for fs view file slice version filtering.

Criteria for different query ranges:

Query Range	File selection criteria	Instant filtering predicate applied to selected files
[earliest, +INF]	The latest snapshot files from table metadata	_
[earliest, endTime]	The latest snapshot files from table metadata	'_hoodie_commit_time' in setA, setA contains the begin instant times for actions completed before or on 'endTime'
[-INF, +INF]	The latest completed instant metadata	'_hoodie_commit_time' = i_n, i_n is the latest completed instant
[-INF, endTime]	I) find the last completed instant i_n before or on 'endTime; II) read the latest snapshot from table metadata if i_n is archived or the commit metadata if it is still active	'_hoodie_commit_time' = i_n
[startTime, +INF]	i).find the instant set setA, setA is a collection of all the instants completed after or on 'startTime'; ii). read the latest snapshot from table metadata if setA has archived instants or the commit metadata if all the instants are still active	'_hoodie_commit_time' in setA
[earliest, endTime]	i).find the instant set setA, setA is a collection of all the instants completed in the given time range; ii). read the latest snapshot from table metadata if setA has archived instants or the commit metadata if all the instants are still active	'_hoodie_commit_time' in setA

A RangeType is required for analyzing the query so that the query range boundary inclusiveness have clear semantics.

IMPORTANT: the reader may optionally choose to fall back to reading the latest snapshot if there are files decoding the commit metadata are already cleaned.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IncrementalQueryAnalyzer.Builder Builder for IncrementalQueryAnalyzer.
static class	IncrementalQueryAnalyzer.QueryContext Represents the analyzed query context.

Field Summary

Fields

Modifier and Type	Field and Description
static String	START_COMMIT_EARLIEST

Method Summary

Modifier and Type	Method and Description
IncrementalQueryAnalyzer.QueryContext	analyze() Analyzes the incremental query context with given completion time range.
static IncrementalQueryAnalyzer.Builder	builder() Returns a builder.
static HoodieTimeline	filterInstantsAsPerUserConfigs(HoodieTableMetaClient metaClient, HoodieTimeline timeline, boolean skipCompaction, boolean skipClustering, boolean skipInsertOverwrite) Filters out the unnecessary instants as per user specified configs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

START_COMMIT_EARLIEST

public static final String START_COMMIT_EARLIEST

See Also:

Constant Field Values

Method Detail

builder

public static IncrementalQueryAnalyzer.Builder builder()

Returns a builder.

analyze

public IncrementalQueryAnalyzer.QueryContext analyze()

Analyzes the incremental query context with given completion time range.

Returns:

An incremental query context including the instant time range info.

filterInstantsAsPerUserConfigs

public static HoodieTimeline filterInstantsAsPerUserConfigs(HoodieTableMetaClient metaClient,
                                                            HoodieTimeline timeline,
                                                            boolean skipCompaction,
                                                            boolean skipClustering,
                                                            boolean skipInsertOverwrite)

Filters out the unnecessary instants as per user specified configs.

Parameters:

timeline - The timeline.

Returns:

the filtered timeline