org.apache.hudi.metadata

Class HoodieMetadataFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.AbstractTableFileSystemView

org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

org.apache.hudi.common.table.view.HoodieTableFileSystemView

org.apache.hudi.metadata.HoodieMetadataFileSystemView

All Implemented Interfaces:

Serializable, AutoCloseable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class HoodieMetadataFileSystemView
extends HoodieTableFileSystemView

HoodieTableFileSystemView implementation that retrieved partition listings from the Metadata Table.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields inherited from class org.apache.hudi.common.table.view.HoodieTableFileSystemView

fgIdToBootstrapBaseFile, fgIdToPendingClustering, fgIdToPendingCompaction, fgIdToPendingLogCompaction, fgIdToReplaceInstants, partitionToFileGroupsMap

Fields inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

completionTimeQueryView, metaClient, readLock, writeLock

Constructor Summary

Constructors

Constructor and Description
HoodieMetadataFileSystemView(HoodieEngineContext engineContext, HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, HoodieMetadataConfig metadataConfig)
HoodieMetadataFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, HoodieTableMetadata tableMetadata)

Method Summary

Modifier and Type	Method and Description
void	close() Allow View to release resources and close.
protected List<String>	getAllPartitionPaths()
protected List<StoragePathInfo>	listPartition(StoragePath partitionPath) Return all the files in the partition by reading from the Metadata Table.
protected Map<Pair<String,StoragePath>,List<StoragePathInfo>>	listPartitions(List<Pair<String,StoragePath>> partitionPathList)
void	reset() Clears the partition Map and reset view states.
void	sync() Syncs the file system view from storage to memory.

Methods inherited from class org.apache.hudi.common.table.view.HoodieTableFileSystemView

addPendingCompactionOperations, addPendingLogCompactionOperations, addReplacedFileGroups, createFileIdToBootstrapBaseFileMap, createFileIdToPendingClusteringMap, createFileIdToPendingCompactionMap, createFileIdToPendingLogCompactionMap, createFileIdToReplaceInstantMap, createPartitionToFileGroups, fetchAllStoredFileGroups, fetchFileGroupsInPendingClustering, getAllFileGroups, getBootstrapBaseFile, getPendingClusteringInstant, getPendingCompactionOperationWithInstant, getPendingLogCompactionOperationWithInstant, getReplaceInstant, hasReplacedFilesInPartition, init, init, isBootstrapBaseFilePresentForFileId, isClosed, isPartitionAvailableInStore, isPendingClusteringScheduledForFileId, isPendingCompactionScheduledForFileId, isPendingLogCompactionScheduledForFileId, removePendingCompactionOperations, removePendingLogCompactionOperations, removeReplacedFileIdsAtInstants, resetPendingCompactionOperations, resetPendingLogCompactionOperations, resetReplacedFileGroups, resetViewState, storePartitionView

Methods inherited from class org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

applyDeltaFileSlicesToPartitionView, getTimeline, maySyncIncrementally, refreshTimeline

Methods inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addFilesToView, addFilesToView, buildFileGroups, buildFileGroups, clear, ensurePartitionLoadedCorrectly, fetchLatestBaseFile, fetchLatestBaseFiles, fetchLatestFileSlice, getAllBaseFiles, getAllFileGroups, getAllFileGroupsStateless, getAllFileSlices, getAllLatestBaseFilesBeforeOrOn, getAllLatestFileSlicesBeforeOrOn, getAllLogsMergedFileSliceBeforeOrOn, getAllReplacedFileGroups, getBaseFileOn, getCompletionTime, getFileGroupsInPendingClustering, getLastInstant, getLatestBaseFile, getLatestBaseFile, getLatestBaseFiles, getLatestBaseFiles, getLatestBaseFilesBeforeOrOn, getLatestBaseFilesInRange, getLatestFileSlice, getLatestFileSliceInRange, getLatestFileSlices, getLatestFileSlicesBeforeOrOn, getLatestFileSlicesIncludingInflight, getLatestFileSlicesStateless, getLatestMergedFileSlicesBeforeOrOn, getLatestUnCompactedFileSlices, getPartitionNames, getPartitionPaths, getPendingCompactionOperations, getPendingLogCompactionOperations, getReplacedFileGroupsAfterOrOn, getReplacedFileGroupsBefore, getReplacedFileGroupsBeforeOrOn, getVisibleCommitsAndCompactionTimeline, isBaseFileDueToPendingClustering, isBaseFileDueToPendingCompaction, loadAllPartitions, loadPartitions, refreshCompletionTimeQueryView

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieMetadataFileSystemView

public HoodieMetadataFileSystemView(HoodieTableMetaClient metaClient,
                                    HoodieTimeline visibleActiveTimeline,
                                    HoodieTableMetadata tableMetadata)

HoodieMetadataFileSystemView

public HoodieMetadataFileSystemView(HoodieEngineContext engineContext,
                                    HoodieTableMetaClient metaClient,
                                    HoodieTimeline visibleActiveTimeline,
                                    HoodieMetadataConfig metadataConfig)

Method Detail

listPartition

protected List<StoragePathInfo> listPartition(StoragePath partitionPath)
                                       throws IOException

Return all the files in the partition by reading from the Metadata Table.

Overrides:

listPartition in class AbstractTableFileSystemView

Parameters:

partitionPath - The absolute path of the partition

Throws:

IOException

getAllPartitionPaths

protected List<String> getAllPartitionPaths()
                                     throws IOException

Overrides:

getAllPartitionPaths in class AbstractTableFileSystemView

Returns:

A list of relative partition paths of all partitions.

Throws:

IOException - upon error.

listPartitions

protected Map<Pair<String,StoragePath>,List<StoragePathInfo>> listPartitions(List<Pair<String,StoragePath>> partitionPathList)
                                                                      throws IOException

Overrides:

listPartitions in class AbstractTableFileSystemView

Parameters:

partitionPathList - A list of pairs of the relative and absolute paths of the partitions.

Returns:

all the files from the partitions.

Throws:

IOException - upon error.

reset

public void reset()

Description copied from class: AbstractTableFileSystemView

Clears the partition Map and reset view states.

NOTE: The logic MUST BE guarded by the write lock.

Specified by:

reset in interface SyncableFileSystemView

Overrides:

reset in class AbstractTableFileSystemView

sync

public void sync()

Description copied from class: AbstractTableFileSystemView

Syncs the file system view from storage to memory. Performs complete reset of file-system view. Subsequent partition view calls will load file slices against the latest timeline.

NOTE: The logic MUST BE guarded by the write lock.

Specified by:

sync in interface SyncableFileSystemView

Overrides:

sync in class IncrementalTimelineSyncFileSystemView

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface SyncableFileSystemView

Overrides:

close in class HoodieTableFileSystemView