org.apache.hudi.common.table.timeline

Class HoodieArchivedTimeline

java.lang.Object

org.apache.hudi.common.table.timeline.HoodieDefaultTimeline

org.apache.hudi.common.table.timeline.HoodieArchivedTimeline

All Implemented Interfaces:

Serializable, HoodieTimeline

public class HoodieArchivedTimeline
extends HoodieDefaultTimeline

Represents the Archived Timeline for the Hoodie table.

Timeline Refresh

Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call #reload().

Serialization/De-serialization

This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieArchivedTimeline.ClosedOpenTimeRangeFilter A time based filter with range [startTs, endTs).
static class	HoodieArchivedTimeline.LoadMode Different mode for loading the archived instant metadata.
static class	HoodieArchivedTimeline.StartTsFilter A time based filter with range [startTs, +∞).
static class	HoodieArchivedTimeline.TimeRangeFilter A time based filter with range (startTs, endTs].

Field Summary

Fields

Modifier and Type	Field and Description
static String	COMPLETION_TIME_ARCHIVED_META_FIELD
static String	INSTANT_TIME_ARCHIVED_META_FIELD

Fields inherited from class org.apache.hudi.common.table.timeline.HoodieDefaultTimeline

details

Fields inherited from interface org.apache.hudi.common.table.timeline.HoodieTimeline

CLEAN_ACTION, CLEAN_EXTENSION, COMMIT_ACTION, COMMIT_EXTENSION, COMPACTION_ACTION, COMPLETED_COMPACTION_SUFFIX, COMPLETED_EXTENSION, DELTA_COMMIT_ACTION, DELTA_COMMIT_EXTENSION, EQUALS, FULL_BOOTSTRAP_INSTANT_TS, GREATER_THAN, GREATER_THAN_OR_EQUALS, INDEX_COMMIT_EXTENSION, INDEXING_ACTION, INFLIGHT_CLEAN_EXTENSION, INFLIGHT_COMMIT_EXTENSION, INFLIGHT_COMPACTION_EXTENSION, INFLIGHT_DELTA_COMMIT_EXTENSION, INFLIGHT_EXTENSION, INFLIGHT_INDEX_COMMIT_EXTENSION, INFLIGHT_LOG_COMPACTION_EXTENSION, INFLIGHT_REPLACE_COMMIT_EXTENSION, INFLIGHT_RESTORE_EXTENSION, INFLIGHT_ROLLBACK_EXTENSION, INFLIGHT_SAVE_SCHEMA_ACTION_EXTENSION, INFLIGHT_SAVEPOINT_EXTENSION, INIT_INSTANT_TS, INVALID_INSTANT_TS, LESSER_THAN, LESSER_THAN_OR_EQUALS, LOG_COMPACTION_ACTION, METADATA_BOOTSTRAP_INSTANT_TS, REPLACE_COMMIT_ACTION, REPLACE_COMMIT_EXTENSION, REQUESTED_CLEAN_EXTENSION, REQUESTED_COMMIT_EXTENSION, REQUESTED_COMPACTION_EXTENSION, REQUESTED_COMPACTION_SUFFIX, REQUESTED_DELTA_COMMIT_EXTENSION, REQUESTED_EXTENSION, REQUESTED_INDEX_COMMIT_EXTENSION, REQUESTED_LOG_COMPACTION_EXTENSION, REQUESTED_LOG_COMPACTION_SUFFIX, REQUESTED_REPLACE_COMMIT_EXTENSION, REQUESTED_RESTORE_EXTENSION, REQUESTED_ROLLBACK_EXTENSION, REQUESTED_SAVE_SCHEMA_ACTION_EXTENSION, RESTORE_ACTION, RESTORE_EXTENSION, ROLLBACK_ACTION, ROLLBACK_EXTENSION, SAVE_SCHEMA_ACTION_EXTENSION, SAVEPOINT_ACTION, SAVEPOINT_EXTENSION, SCHEMA_COMMIT_ACTION, VALID_ACTIONS_IN_TIMELINE

Constructor Summary

Constructors

Constructor and Description
HoodieArchivedTimeline() Deprecated.
HoodieArchivedTimeline(HoodieTableMetaClient metaClient) Loads all the archived instants.
HoodieArchivedTimeline(HoodieTableMetaClient metaClient, String startTs) Loads completed instants from startTs(inclusive).

Method Summary

Modifier and Type	Method and Description
void	clearInstantDetailsFromMemory(String instantTime)
void	clearInstantDetailsFromMemory(String startTs, String endTs)
Option<byte[]>	getInstantDetails(HoodieInstant instant) Read the completed instant details.
HoodieDefaultTimeline	getWriteTimeline() Timeline to just include commits (commit/deltacommit), compaction and replace actions.
void	loadCompactionDetailsInMemory(String compactionInstantTime)
void	loadCompactionDetailsInMemory(String startTs, String endTs)
void	loadCompletedInstantDetailsInMemory()
void	loadInstantDetailsInMemory(String startTs, String endTs)
static void	loadInstants(HoodieTableMetaClient metaClient, HoodieArchivedTimeline.TimeRangeFilter filter, HoodieArchivedTimeline.LoadMode loadMode, Function<org.apache.avro.generic.GenericRecord,Boolean> commitsFilter, BiConsumer<String,org.apache.avro.generic.GenericRecord> recordConsumer) Loads the instants from the timeline.
HoodieArchivedTimeline	reload()
HoodieArchivedTimeline	reload(String startTs) Reloads the archived timeline incrementally with given beginning timestamp startTs.

Methods inherited from class org.apache.hudi.common.table.timeline.HoodieDefaultTimeline

appendInstants, containsInstant, containsInstant, containsOrBeforeTimelineStarts, countInstants, empty, filter, filterCompletedAndCompactionInstants, filterCompletedIndexTimeline, filterCompletedInstants, filterCompletedInstantsOrRewriteTimeline, filterCompletedOrMajorOrMinorCompactionInstants, filterInflights, filterInflightsAndRequested, filterInstantsByAction, filterPendingCompactionTimeline, filterPendingExcludingCompaction, filterPendingExcludingLogCompaction, filterPendingExcludingMajorAndMinorCompaction, filterPendingIndexTimeline, filterPendingLogCompactionTimeline, filterPendingMajorOrMinorCompactionTimeline, filterPendingReplaceTimeline, filterPendingRollbackTimeline, filterRequestedRollbackTimeline, findInstantBefore, findInstantsAfter, findInstantsAfter, findInstantsAfterOrEquals, findInstantsBefore, findInstantsBeforeOrEquals, findInstantsInClosedRange, findInstantsInRange, findInstantsInRangeByCompletionTime, findInstantsModifiedAfterByCompletionTime, firstInstant, firstInstant, getAllCommitsTimeline, getCleanerTimeline, getCommitAndReplaceTimeline, getCommitsAndCompactionTimeline, getCommitsTimeline, getCommitTimeline, getCompletedReplaceTimeline, getContiguousCompletedWriteTimeline, getDeltaCommitTimeline, getFirstNonSavepointCommit, getFirstPendingClusterInstant, getInstants, getInstantsAsStream, getInstantsOrderedByCompletionTime, getLastClusteringInstant, getLastPendingClusterInstant, getRestoreTimeline, getReverseOrderedInstants, getRollbackAndRestoreTimeline, getRollbackTimeline, getSavePointTimeline, getTimelineHash, getTimelineOfActions, isBeforeTimelineStarts, isEmpty, isPendingClusterInstant, lastInstant, mergeTimeline, nthFromLastInstant, nthInstant, setInstants, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.table.timeline.HoodieTimeline

compareTimestamps, getCleanInflightInstant, getCleanRequestedInstant, getCommitFromCommitFile, getCompactionInflightInstant, getCompactionRequestedInstant, getIndexInflightInstant, getIndexRequestedInstant, getInflightInstant, getLogCompactionInflightInstant, getLogCompactionRequestedInstant, getReplaceCommitInflightInstant, getReplaceCommitRequestedInstant, getRequestedInstant, getRestoreRequestedInstant, getRollbackRequestedInstant, isInClosedOpenRange, isInClosedRange, isInRange, makeCleanerFileName, makeCommitFileName, makeDeltaFileName, makeFileNameAsComplete, makeFileNameAsInflight, makeIndexCommitFileName, makeInflightCleanerFileName, makeInflightCommitFileName, makeInflightCompactionFileName, makeInflightDeltaFileName, makeInflightIndexFileName, makeInflightLogCompactionFileName, makeInflightReplaceFileName, makeInflightRestoreFileName, makeInflightRollbackFileName, makeInflightSavePointFileName, makeInflightSchemaFileName, makeReplaceFileName, makeRequestedCleanerFileName, makeRequestedCommitFileName, makeRequestedCompactionFileName, makeRequestedDeltaFileName, makeRequestedIndexFileName, makeRequestedLogCompactionFileName, makeRequestedReplaceFileName, makeRequestedRestoreFileName, makeRequestedRollbackFileName, makeRequestSchemaFileName, makeRestoreFileName, makeRollbackFileName, makeSavePointFileName, makeSchemaFileName, maxInstant, minInstant

Field Detail

INSTANT_TIME_ARCHIVED_META_FIELD

public static final String INSTANT_TIME_ARCHIVED_META_FIELD

See Also:

Constant Field Values

COMPLETION_TIME_ARCHIVED_META_FIELD

public static final String COMPLETION_TIME_ARCHIVED_META_FIELD

See Also:

Constant Field Values

Constructor Detail

HoodieArchivedTimeline

public HoodieArchivedTimeline(HoodieTableMetaClient metaClient)

Loads all the archived instants. Note that there is no lazy loading, so this may not work if the archived timeline range is really long. TBD: Should we enforce maximum time range?

HoodieArchivedTimeline

public HoodieArchivedTimeline(HoodieTableMetaClient metaClient,
                              String startTs)

Loads completed instants from startTs(inclusive). Note that there is no lazy loading, so this may not work if really early startTs is specified.

HoodieArchivedTimeline

public HoodieArchivedTimeline()

Deprecated.

For serialization and de-serialization only.

Method Detail

loadInstantDetailsInMemory

public void loadInstantDetailsInMemory(String startTs,
                                       String endTs)

loadCompletedInstantDetailsInMemory

public void loadCompletedInstantDetailsInMemory()

loadCompactionDetailsInMemory

public void loadCompactionDetailsInMemory(String compactionInstantTime)

loadCompactionDetailsInMemory

public void loadCompactionDetailsInMemory(String startTs,
                                          String endTs)

clearInstantDetailsFromMemory

public void clearInstantDetailsFromMemory(String instantTime)

clearInstantDetailsFromMemory

public void clearInstantDetailsFromMemory(String startTs,
                                          String endTs)

getInstantDetails

public Option<byte[]> getInstantDetails(HoodieInstant instant)

Description copied from interface: HoodieTimeline

Read the completed instant details.

Specified by:

getInstantDetails in interface HoodieTimeline

Overrides:

getInstantDetails in class HoodieDefaultTimeline

reload

public HoodieArchivedTimeline reload()

reload

public HoodieArchivedTimeline reload(String startTs)

Reloads the archived timeline incrementally with given beginning timestamp startTs. This method is not thread safe.

IMPORTANT: this is for multiple loading of one static snapshot of the timeline, if there is new instants got archived, use reload() instead.

loadInstants

public static void loadInstants(HoodieTableMetaClient metaClient,
                                @Nullable
                                HoodieArchivedTimeline.TimeRangeFilter filter,
                                HoodieArchivedTimeline.LoadMode loadMode,
                                Function<org.apache.avro.generic.GenericRecord,Boolean> commitsFilter,
                                BiConsumer<String,org.apache.avro.generic.GenericRecord> recordConsumer)

Loads the instants from the timeline.

Parameters:

metaClient - The meta client.

filter - The time range filter where the target instant belongs to.

loadMode - The load mode.

commitsFilter - Filter of the instant type.

recordConsumer - Consumer of the instant record payload.

getWriteTimeline

public HoodieDefaultTimeline getWriteTimeline()

Description copied from interface: HoodieTimeline

Timeline to just include commits (commit/deltacommit), compaction and replace actions.

Specified by:

getWriteTimeline in interface HoodieTimeline

Overrides:

getWriteTimeline in class HoodieDefaultTimeline

Returns: