org.apache.hudi.common.table.view

Class RocksDbBasedFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.AbstractTableFileSystemView

org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

org.apache.hudi.common.table.view.RocksDbBasedFileSystemView

All Implemented Interfaces:

Serializable, AutoCloseable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class RocksDbBasedFileSystemView
extends IncrementalTimelineSyncFileSystemView

A file-system view implementation on top of embedded Rocks DB store. For each table : 3 column Family is added for storing (1) File-Slices and Data Files for View lookups (2) Pending compaction operations (3) Partitions tracked Fine-grained retrieval API to fetch latest file-slice and data-file which are common operations for ingestion/compaction are supported. TODO: vb The current implementation works in embedded server mode where each restarts blows away the view stores. To support view-state preservation across restarts, Hoodie timeline also needs to be stored inorder to detect changes to timeline across restarts.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

completionTimeQueryView, metaClient, readLock, writeLock

Constructor Summary

Constructors

Constructor and Description
RocksDbBasedFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, FileSystemViewStorageConfig config)
RocksDbBasedFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline, List<StoragePathInfo> pathInfoList, FileSystemViewStorageConfig config)

Method Summary

Modifier and Type	Method and Description
protected void	addPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) Add pending compaction operations to store.
protected void	addPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) Add pending Log compaction operations to store.
protected void	addReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups) Track instant time for new file groups replaced.
protected void	applyDeltaFileSlicesToPartitionView(String partition, List<HoodieFileGroup> deltaFileGroups, org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView.DeltaApplyMode mode) Apply changes to partition file-system view.
void	close() Allow View to release resources and close.
Stream<Pair<HoodieFileGroupId,HoodieInstant>>	fetchFileGroupsInPendingClustering() Fetch all file groups in pending clustering.
protected Option<HoodieBaseFile>	fetchLatestBaseFile(String partitionPath, String fileId) Default implementation for fetching latest base-file.
Option<FileSlice>	fetchLatestFileSlice(String partitionPath, String fileId) Default implementation for fetching file-slice.
Option<FileSlice>	fetchLatestFileSliceBeforeOrOn(String partitionPath, String fileId, String instantTime)
protected Option<BootstrapBaseFileMapping>	getBootstrapBaseFile(HoodieFileGroupId fileGroupId) Return pending compaction operation for a file-group.
protected Option<HoodieInstant>	getPendingClusteringInstant(HoodieFileGroupId fgId) Get pending clustering instant time for specified file group.
protected Option<Pair<String,CompactionOperation>>	getPendingCompactionOperationWithInstant(HoodieFileGroupId fgId) Return pending compaction operation for a file-group.
protected Option<Pair<String,CompactionOperation>>	getPendingLogCompactionOperationWithInstant(HoodieFileGroupId fgId) Return pending Log compaction operation for a file-group.
protected Option<HoodieInstant>	getReplaceInstant(HoodieFileGroupId fileGroupId) Track instant time for file groups replaced.
protected boolean	hasReplacedFilesInPartition(String partitionPath) Returns whether there are replaced files within the given partition.
protected void	init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline) Initialize the view.
protected boolean	isBootstrapBaseFilePresentForFileId(HoodieFileGroupId fgId) Check if there is a bootstrap base file present for this file.
protected boolean	isPartitionAvailableInStore(String partitionPath) Checks if partition is pre-loaded and available in store.
protected boolean	isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
protected boolean	isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding compaction scheduled for this file.
protected boolean	isPendingLogCompactionScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding log compaction scheduled for this file.
protected void	removeReplacedFileIdsAtInstants(Set<String> instants) Remove file groups that are replaced in any of the specified instants.
protected void	resetPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) resets the pending compaction operation and overwrite with the new list.
protected void	resetPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations) resets the pending Log compaction operation and overwrite with the new list.
protected void	resetReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups) Track instant time for file groups replaced.
protected void	resetViewState() Allows all view metadata in file system view storage to be reset by subclasses.
protected void	storePartitionView(String partitionPath, List<HoodieFileGroup> fileGroups) Add a complete partition view to store.

Methods inherited from class org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView

getTimeline, maySyncIncrementally, refreshTimeline, sync

Methods inherited from class org.apache.hudi.common.table.view.AbstractTableFileSystemView

addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addBootstrapBaseFileIfPresent, addFilesToView, addFilesToView, buildFileGroups, buildFileGroups, clear, ensurePartitionLoadedCorrectly, fetchLatestBaseFiles, getAllBaseFiles, getAllFileGroups, getAllFileGroupsStateless, getAllFileSlices, getAllLatestBaseFilesBeforeOrOn, getAllLatestFileSlicesBeforeOrOn, getAllLogsMergedFileSliceBeforeOrOn, getAllPartitionPaths, getAllReplacedFileGroups, getBaseFileOn, getCompletionTime, getFileGroupsInPendingClustering, getLastInstant, getLatestBaseFile, getLatestBaseFile, getLatestBaseFiles, getLatestBaseFiles, getLatestBaseFilesBeforeOrOn, getLatestBaseFilesInRange, getLatestFileSlice, getLatestFileSliceInRange, getLatestFileSlices, getLatestFileSlicesBeforeOrOn, getLatestFileSlicesIncludingInflight, getLatestFileSlicesStateless, getLatestMergedFileSlicesBeforeOrOn, getLatestUnCompactedFileSlices, getPartitionNames, getPartitionPaths, getPendingCompactionOperations, getPendingLogCompactionOperations, getReplacedFileGroupsAfterOrOn, getReplacedFileGroupsBefore, getReplacedFileGroupsBeforeOrOn, getVisibleCommitsAndCompactionTimeline, isBaseFileDueToPendingClustering, isBaseFileDueToPendingCompaction, listPartition, listPartitions, loadAllPartitions, loadPartitions, refreshCompletionTimeQueryView, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RocksDbBasedFileSystemView

public RocksDbBasedFileSystemView(HoodieTableMetaClient metaClient,
                                  HoodieTimeline visibleActiveTimeline,
                                  FileSystemViewStorageConfig config)

RocksDbBasedFileSystemView

public RocksDbBasedFileSystemView(HoodieTableMetaClient metaClient,
                                  HoodieTimeline visibleActiveTimeline,
                                  List<StoragePathInfo> pathInfoList,
                                  FileSystemViewStorageConfig config)

Method Detail

init

protected void init(HoodieTableMetaClient metaClient,
                    HoodieTimeline visibleActiveTimeline)

Description copied from class: AbstractTableFileSystemView

Initialize the view.

Overrides:

init in class AbstractTableFileSystemView

isPendingCompactionScheduledForFileId

protected boolean isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an outstanding compaction scheduled for this file.

Specified by:

isPendingCompactionScheduledForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending compaction, false otherwise

resetPendingCompactionOperations

protected void resetPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

resets the pending compaction operation and overwrite with the new list.

Parameters:

operations - Pending Compaction Operations

addPendingCompactionOperations

protected void addPendingCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

Add pending compaction operations to store.

Parameters:

operations - Pending compaction operations to be added

isPendingLogCompactionScheduledForFileId

protected boolean isPendingLogCompactionScheduledForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an outstanding log compaction scheduled for this file.

Specified by:

isPendingLogCompactionScheduledForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending log compaction, false otherwise

resetPendingLogCompactionOperations

protected void resetPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

resets the pending Log compaction operation and overwrite with the new list.

Parameters:

operations - Pending Log Compaction Operations

addPendingLogCompactionOperations

protected void addPendingLogCompactionOperations(Stream<Pair<String,CompactionOperation>> operations)

Description copied from class: AbstractTableFileSystemView

Add pending Log compaction operations to store.

Parameters:

operations - Pending Log compaction operations to be added

isPendingClusteringScheduledForFileId

protected boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.

Specified by:

isPendingClusteringScheduledForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending clustering, false otherwise

getPendingClusteringInstant

protected Option<HoodieInstant> getPendingClusteringInstant(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Get pending clustering instant time for specified file group. Return None if file group is not in pending clustering operation.

Specified by:

getPendingClusteringInstant in class AbstractTableFileSystemView

fetchFileGroupsInPendingClustering

public Stream<Pair<HoodieFileGroupId,HoodieInstant>> fetchFileGroupsInPendingClustering()

Description copied from class: AbstractTableFileSystemView

Fetch all file groups in pending clustering.

Specified by:

fetchFileGroupsInPendingClustering in class AbstractTableFileSystemView

resetViewState

protected void resetViewState()

Description copied from class: AbstractTableFileSystemView

Allows all view metadata in file system view storage to be reset by subclasses.

Specified by:

resetViewState in class AbstractTableFileSystemView

getPendingCompactionOperationWithInstant

protected Option<Pair<String,CompactionOperation>> getPendingCompactionOperationWithInstant(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Return pending compaction operation for a file-group.

Specified by:

getPendingCompactionOperationWithInstant in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

getPendingLogCompactionOperationWithInstant

protected Option<Pair<String,CompactionOperation>> getPendingLogCompactionOperationWithInstant(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Return pending Log compaction operation for a file-group.

Specified by:

getPendingLogCompactionOperationWithInstant in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

isPartitionAvailableInStore

protected boolean isPartitionAvailableInStore(String partitionPath)

Description copied from class: AbstractTableFileSystemView

Checks if partition is pre-loaded and available in store.

Parameters:

partitionPath - Partition Path

storePartitionView

protected void storePartitionView(String partitionPath,
                                  List<HoodieFileGroup> fileGroups)

Description copied from class: AbstractTableFileSystemView

Add a complete partition view to store.

Parameters:

partitionPath - Partition Path

fileGroups - File Groups for the partition path

applyDeltaFileSlicesToPartitionView

protected void applyDeltaFileSlicesToPartitionView(String partition,
                                                   List<HoodieFileGroup> deltaFileGroups,
                                                   org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView.DeltaApplyMode mode)

Description copied from class: IncrementalTimelineSyncFileSystemView

Apply changes to partition file-system view. Base Implementation overwrites the entire partitions view assuming some sort of map (in-mem/disk-based) is used. For View implementation which supports fine-granular updates (e:g RocksDB), override this method.

Overrides:

applyDeltaFileSlicesToPartitionView in class IncrementalTimelineSyncFileSystemView

Parameters:

partition - PartitionPath

deltaFileGroups - Changed file-slices aggregated as file-groups

mode - Delta Apply mode

isBootstrapBaseFilePresentForFileId

protected boolean isBootstrapBaseFilePresentForFileId(HoodieFileGroupId fgId)

Description copied from class: AbstractTableFileSystemView

Check if there is a bootstrap base file present for this file.

Specified by:

isBootstrapBaseFilePresentForFileId in class AbstractTableFileSystemView

Parameters:

fgId - File-Group Id

Returns:

true if there is associated bootstrap base-file, false otherwise

getBootstrapBaseFile

protected Option<BootstrapBaseFileMapping> getBootstrapBaseFile(HoodieFileGroupId fileGroupId)

Description copied from class: AbstractTableFileSystemView

Return pending compaction operation for a file-group.

Specified by:

getBootstrapBaseFile in class AbstractTableFileSystemView

Parameters:

fileGroupId - File-Group Id

fetchLatestFileSlice

public Option<FileSlice> fetchLatestFileSlice(String partitionPath,
                                              String fileId)

Description copied from class: AbstractTableFileSystemView

Default implementation for fetching file-slice.

Overrides:

fetchLatestFileSlice in class AbstractTableFileSystemView

Parameters:

partitionPath - Partition path

fileId - File Id

Returns:

File Slice if present

fetchLatestFileSliceBeforeOrOn

public Option<FileSlice> fetchLatestFileSliceBeforeOrOn(String partitionPath,
                                                        String fileId,
                                                        String instantTime)

fetchLatestBaseFile

protected Option<HoodieBaseFile> fetchLatestBaseFile(String partitionPath,
                                                     String fileId)

Description copied from class: AbstractTableFileSystemView

Default implementation for fetching latest base-file.

Overrides:

fetchLatestBaseFile in class AbstractTableFileSystemView

Parameters:

partitionPath - Partition path

fileId - File Id

Returns:

base File if present

resetReplacedFileGroups

protected void resetReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Description copied from class: AbstractTableFileSystemView

Track instant time for file groups replaced.

Specified by:

resetReplacedFileGroups in class AbstractTableFileSystemView

addReplacedFileGroups

protected void addReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Description copied from class: AbstractTableFileSystemView

Track instant time for new file groups replaced.

Specified by:

addReplacedFileGroups in class AbstractTableFileSystemView

removeReplacedFileIdsAtInstants

protected void removeReplacedFileIdsAtInstants(Set<String> instants)

Description copied from class: AbstractTableFileSystemView

Remove file groups that are replaced in any of the specified instants.

Specified by:

removeReplacedFileIdsAtInstants in class AbstractTableFileSystemView

hasReplacedFilesInPartition

protected boolean hasReplacedFilesInPartition(String partitionPath)

Description copied from class: AbstractTableFileSystemView

Returns whether there are replaced files within the given partition.

Specified by:

hasReplacedFilesInPartition in class AbstractTableFileSystemView

getReplaceInstant

protected Option<HoodieInstant> getReplaceInstant(HoodieFileGroupId fileGroupId)

Description copied from class: AbstractTableFileSystemView

Track instant time for file groups replaced.

Specified by:

getReplaceInstant in class AbstractTableFileSystemView

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface SyncableFileSystemView

Overrides:

close in class AbstractTableFileSystemView