org.apache.hudi.common.table.view

Class AbstractTableFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.AbstractTableFileSystemView

All Implemented Interfaces:

Serializable, AutoCloseable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Direct Known Subclasses:

IncrementalTimelineSyncFileSystemView

public abstract class AbstractTableFileSystemView
extends Object
implements SyncableFileSystemView, Serializable

Common thread-safe implementation for multiple TableFileSystemView Implementations. Provides uniform handling of:

• Loading file-system views from underlying file-system;
• Pending compaction operations and changing file-system views based on that;
• Thread-safety in loading and managing file system views for this table;
• resetting file-system views.

The actual mechanism of fetching file slices from different view storages is delegated to sub-classes.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields

Modifier and Type	Field and Description
protected CompletionTimeQueryView	completionTimeQueryView
protected HoodieTableMetaClient	metaClient
protected ReentrantReadWriteLock.ReadLock	readLock
protected ReentrantReadWriteLock.WriteLock	writeLock

Constructor Summary

Constructors

Constructor and Description
AbstractTableFileSystemView()

Method Summary

Modifier and Type	Method and Description
protected FileSlice	addBootstrapBaseFileIfPresent(FileSlice fileSlice)
protected FileSlice	addBootstrapBaseFileIfPresent(FileSlice fileSlice, Function<HoodieFileGroupId,Option<BootstrapBaseFileMapping>> bootstrapBaseFileMappingFunc)
protected HoodieFileGroup	addBootstrapBaseFileIfPresent(HoodieFileGroup fileGroup)
protected HoodieFileGroup	addBootstrapBaseFileIfPresent(HoodieFileGroup fileGroup, Function<HoodieFileGroupId,Option<BootstrapBaseFileMapping>> bootstrapBaseFileMappingFunc)
protected HoodieBaseFile	addBootstrapBaseFileIfPresent(HoodieFileGroupId fileGroupId, HoodieBaseFile baseFile)
protected HoodieBaseFile	addBootstrapBaseFileIfPresent(HoodieFileGroupId fileGroupId, HoodieBaseFile baseFile, Function<HoodieFileGroupId,Option<BootstrapBaseFileMapping>> bootstrapBaseFileMappingFunc)
List<HoodieFileGroup>	addFilesToView(List<StoragePathInfo> statuses) Adds the provided statuses into the file system view, and also caches it inside this object.
List<HoodieFileGroup>	addFilesToView(String partitionPath, List<StoragePathInfo> statuses) Adds the provided statuses into the file system view for a single partition, and also caches it inside this object.
protected abstract void	addReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups) Track instant time for new file groups replaced.
protected List<HoodieFileGroup>	buildFileGroups(String partition, List<StoragePathInfo> statuses, HoodieTimeline timeline, boolean addPendingCompactionFileSlice) Build FileGroups from passed in file-status.
protected List<HoodieFileGroup>	buildFileGroups(String partition, Stream<HoodieBaseFile> baseFileStream, Stream<HoodieLogFile> logFileStream, HoodieTimeline timeline, boolean addPendingCompactionFileSlice)
protected void	clear() Clear the resource.
void	close() Allow View to release resources and close.
protected void	ensurePartitionLoadedCorrectly(String partition) Allows lazily loading the partitions if needed.
protected abstract Stream<Pair<HoodieFileGroupId,HoodieInstant>>	fetchFileGroupsInPendingClustering() Fetch all file groups in pending clustering.
protected Option<HoodieBaseFile>	fetchLatestBaseFile(String partitionPath, String fileId) Default implementation for fetching latest base-file.
Stream<HoodieBaseFile>	fetchLatestBaseFiles(String partitionPath) Default implementation for fetching latest base-files for the partition-path.
Option<FileSlice>	fetchLatestFileSlice(String partitionPath, String fileId) Default implementation for fetching file-slice.
Stream<HoodieBaseFile>	getAllBaseFiles(String partitionStr) Stream all the data file versions grouped by FileId for a given partition.
Stream<HoodieFileGroup>	getAllFileGroups(String partitionStr) Stream all the file groups for a given partition.
Stream<HoodieFileGroup>	getAllFileGroupsStateless(String partitionStr) Stream all the file groups for a given partition without caching the file group mappings.
Stream<FileSlice>	getAllFileSlices(String partitionStr) Stream all the file slices for a given partition, latest or not.
Map<String,Stream<HoodieBaseFile>>	getAllLatestBaseFilesBeforeOrOn(String maxCommitTime) Streams the latest version base files in all partitions with precondition that commitTime(file) before maxCommitTime.
Map<String,Stream<FileSlice>>	getAllLatestFileSlicesBeforeOrOn(String maxCommitTime) Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getAllLogsMergedFileSliceBeforeOrOn(String partitionStr, String maxInstantTime) Stream all "merged" file-slices before on an instant time for a MERGE_ON_READ table with index that can index log files(which means it writes pure logs first).
protected List<String>	getAllPartitionPaths()
Stream<HoodieFileGroup>	getAllReplacedFileGroups(String partitionPath) Stream all the replaced file groups for given partition.
Option<HoodieBaseFile>	getBaseFileOn(String partitionStr, String instantTime, String fileId) Get the version of data file matching the instant time in the given partition.
protected abstract Option<BootstrapBaseFileMapping>	getBootstrapBaseFile(HoodieFileGroupId fileGroupId) Return pending compaction operation for a file-group.
Option<String>	getCompletionTime(String instantTime) Returns the completion time for instant.
Stream<Pair<HoodieFileGroupId,HoodieInstant>>	getFileGroupsInPendingClustering() Filegroups that are in pending clustering.
Option<HoodieInstant>	getLastInstant() Last Known Instant on which the view is built.
protected Option<HoodieBaseFile>	getLatestBaseFile(HoodieFileGroup fileGroup)
Option<HoodieBaseFile>	getLatestBaseFile(String partitionStr, String fileId) Get Latest base file for a partition and file-Id.
Stream<HoodieBaseFile>	getLatestBaseFiles() Stream all the latest data files, in the file system view.
Stream<HoodieBaseFile>	getLatestBaseFiles(String partitionStr) Stream all the latest data files in the given partition.
Stream<HoodieBaseFile>	getLatestBaseFilesBeforeOrOn(String partitionStr, String maxCommitTime) Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieBaseFile>	getLatestBaseFilesInRange(List<String> commitsToReturn) Stream all the latest data files pass.
Option<FileSlice>	getLatestFileSlice(String partitionStr, String fileId) Get Latest File Slice for a given fileId in a given partition.
Stream<FileSlice>	getLatestFileSliceInRange(List<String> commitsToReturn) Stream all the latest file slices, in the given range.
Stream<FileSlice>	getLatestFileSlices(String partitionStr) Stream all the latest file slices in the given partition.
Stream<FileSlice>	getLatestFileSlicesBeforeOrOn(String partitionStr, String maxCommitTime, boolean includeFileSlicesInPendingCompaction) Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getLatestFileSlicesIncludingInflight(String partitionPath) Get the latest file slices for a given partition including the inflight ones.
Stream<FileSlice>	getLatestFileSlicesStateless(String partitionStr) Stream all the latest file slices in the given partition without caching the file group mappings.
Stream<FileSlice>	getLatestMergedFileSlicesBeforeOrOn(String partitionStr, String maxInstantTime) Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.
Stream<FileSlice>	getLatestUnCompactedFileSlices(String partitionStr) Stream all the latest uncompacted file slices in the given partition.
List<String>	getPartitionNames()
List<StoragePath>	getPartitionPaths()
protected abstract Option<HoodieInstant>	getPendingClusteringInstant(HoodieFileGroupId fileGroupId) Get pending clustering instant time for specified file group.
Stream<Pair<String,CompactionOperation>>	getPendingCompactionOperations() Return Pending Compaction Operations.
protected abstract Option<Pair<String,CompactionOperation>>	getPendingCompactionOperationWithInstant(HoodieFileGroupId fileGroupId) Return pending compaction operation for a file-group.
Stream<Pair<String,CompactionOperation>>	getPendingLogCompactionOperations() Return Pending Compaction Operations.
protected abstract Option<Pair<String,CompactionOperation>>	getPendingLogCompactionOperationWithInstant(HoodieFileGroupId fileGroupId) Return pending Log compaction operation for a file-group.
Stream<HoodieFileGroup>	getReplacedFileGroupsAfterOrOn(String minCommitTime, String partitionPath) Stream all the replaced file groups after or on minCommitTime.
Stream<HoodieFileGroup>	getReplacedFileGroupsBefore(String maxCommitTime, String partitionPath) Stream all the replaced file groups before maxCommitTime for given partition.
Stream<HoodieFileGroup>	getReplacedFileGroupsBeforeOrOn(String maxCommitTime, String partitionPath) Stream all the replaced file groups before or on maxCommitTime for given partition.
protected abstract Option<HoodieInstant>	getReplaceInstant(HoodieFileGroupId fileGroupId) Track instant time for file groups replaced.
HoodieTimeline	getTimeline() Timeline corresponding to the view.
HoodieTimeline	getVisibleCommitsAndCompactionTimeline() Return Only Commits and Compaction timeline for building file-groups.
protected abstract boolean	hasReplacedFilesInPartition(String partitionPath) Returns whether there are replaced files within the given partition.
protected void	init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline) Initialize the view.
protected boolean	isBaseFileDueToPendingClustering(HoodieBaseFile baseFile) With async clustering, it is possible to see partial/complete base-files due to inflight-clustering, Ignore those base-files.
protected boolean	isBaseFileDueToPendingCompaction(String partitionPath, HoodieBaseFile baseFile) With async compaction, it is possible to see partial/complete base-files due to inflight-compactions, Ignore those base-files.
protected abstract boolean	isBootstrapBaseFilePresentForFileId(HoodieFileGroupId fgId) Check if there is a bootstrap base file present for this file.
protected abstract boolean	isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
protected abstract boolean	isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding compaction scheduled for this file.
protected abstract boolean	isPendingLogCompactionScheduledForFileId(HoodieFileGroupId fgId) Check if there is an outstanding log compaction scheduled for this file.
protected List<StoragePathInfo>	listPartition(StoragePath partitionPath) Return all the files from the partition.
protected Map<Pair<String,StoragePath>,List<StoragePathInfo>>	listPartitions(List<Pair<String,StoragePath>> partitionPathList)
void	loadAllPartitions() Load all partition and file slices into view
void	loadPartitions(List<String> partitionPaths) Load all partition and file slices into view for the provided partition paths
protected void	refreshCompletionTimeQueryView() Refresh the completion time query view.
protected void	refreshTimeline(HoodieTimeline visibleActiveTimeline) Refresh commits timeline.
protected abstract void	removeReplacedFileIdsAtInstants(Set<String> instants) Remove file groups that are replaced in any of the specified instants.
void	reset() Clears the partition Map and reset view states.
protected abstract void	resetReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups) Track instant time for file groups replaced.
protected abstract void	resetViewState() Allows all view metadata in file system view storage to be reset by subclasses.
void	sync() Syncs the file system view from storage to memory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

metaClient

protected HoodieTableMetaClient metaClient

completionTimeQueryView

protected CompletionTimeQueryView completionTimeQueryView

readLock

protected final ReentrantReadWriteLock.ReadLock readLock

writeLock

protected final ReentrantReadWriteLock.WriteLock writeLock

Constructor Detail

AbstractTableFileSystemView

public AbstractTableFileSystemView()

Method Detail

init

protected void init(HoodieTableMetaClient metaClient,
                    HoodieTimeline visibleActiveTimeline)

Initialize the view.

refreshTimeline

protected void refreshTimeline(HoodieTimeline visibleActiveTimeline)

Refresh commits timeline.

Parameters:

visibleActiveTimeline - Visible Active Timeline

refreshCompletionTimeQueryView

protected void refreshCompletionTimeQueryView()

Refresh the completion time query view.

getCompletionTime

public Option<String> getCompletionTime(String instantTime)

Returns the completion time for instant.

addFilesToView

public List<HoodieFileGroup> addFilesToView(List<StoragePathInfo> statuses)

Adds the provided statuses into the file system view, and also caches it inside this object. If the file statuses are limited to a single partition, use addFilesToView(String, List) instead.

addFilesToView

public List<HoodieFileGroup> addFilesToView(String partitionPath,
                                            List<StoragePathInfo> statuses)

Adds the provided statuses into the file system view for a single partition, and also caches it inside this object.

buildFileGroups

protected List<HoodieFileGroup> buildFileGroups(String partition,
                                                List<StoragePathInfo> statuses,
                                                HoodieTimeline timeline,
                                                boolean addPendingCompactionFileSlice)

Build FileGroups from passed in file-status.

buildFileGroups

protected List<HoodieFileGroup> buildFileGroups(String partition,
                                                Stream<HoodieBaseFile> baseFileStream,
                                                Stream<HoodieLogFile> logFileStream,
                                                HoodieTimeline timeline,
                                                boolean addPendingCompactionFileSlice)

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface SyncableFileSystemView

reset

public void reset()

Clears the partition Map and reset view states.

NOTE: The logic MUST BE guarded by the write lock.

Specified by:

reset in interface SyncableFileSystemView

clear

protected void clear()

Clear the resource.

resetViewState

protected abstract void resetViewState()

Allows all view metadata in file system view storage to be reset by subclasses.

getAllPartitionPaths

protected List<String> getAllPartitionPaths()
                                     throws IOException

Returns:

A list of relative partition paths of all partitions.

Throws:

IOException - upon error.

listPartitions

protected Map<Pair<String,StoragePath>,List<StoragePathInfo>> listPartitions(List<Pair<String,StoragePath>> partitionPathList)
                                                                      throws IOException

Parameters:

partitionPathList - A list of pairs of the relative and absolute paths of the partitions.

Returns:

all the files from the partitions.

Throws:

IOException - upon error.

ensurePartitionLoadedCorrectly

protected void ensurePartitionLoadedCorrectly(String partition)

Allows lazily loading the partitions if needed.

Parameters:

partition - partition to be loaded if not present

listPartition

protected List<StoragePathInfo> listPartition(StoragePath partitionPath)
                                       throws IOException

Return all the files from the partition.

Parameters:

partitionPath - The absolute path of the partition

Throws:

IOException

isBaseFileDueToPendingCompaction

protected boolean isBaseFileDueToPendingCompaction(String partitionPath,
                                                   HoodieBaseFile baseFile)

With async compaction, it is possible to see partial/complete base-files due to inflight-compactions, Ignore those base-files.

Parameters:

partitionPath - partition path for the base file

baseFile - base File

isBaseFileDueToPendingClustering

protected boolean isBaseFileDueToPendingClustering(HoodieBaseFile baseFile)

With async clustering, it is possible to see partial/complete base-files due to inflight-clustering, Ignore those base-files.

Parameters:

baseFile - base File

addBootstrapBaseFileIfPresent

protected HoodieFileGroup addBootstrapBaseFileIfPresent(HoodieFileGroup fileGroup)

addBootstrapBaseFileIfPresent

protected HoodieFileGroup addBootstrapBaseFileIfPresent(HoodieFileGroup fileGroup,
                                                        Function<HoodieFileGroupId,Option<BootstrapBaseFileMapping>> bootstrapBaseFileMappingFunc)

addBootstrapBaseFileIfPresent

protected FileSlice addBootstrapBaseFileIfPresent(FileSlice fileSlice)

addBootstrapBaseFileIfPresent

protected FileSlice addBootstrapBaseFileIfPresent(FileSlice fileSlice,
                                                  Function<HoodieFileGroupId,Option<BootstrapBaseFileMapping>> bootstrapBaseFileMappingFunc)

addBootstrapBaseFileIfPresent

protected HoodieBaseFile addBootstrapBaseFileIfPresent(HoodieFileGroupId fileGroupId,
                                                       HoodieBaseFile baseFile)

addBootstrapBaseFileIfPresent

protected HoodieBaseFile addBootstrapBaseFileIfPresent(HoodieFileGroupId fileGroupId,
                                                       HoodieBaseFile baseFile,
                                                       Function<HoodieFileGroupId,Option<BootstrapBaseFileMapping>> bootstrapBaseFileMappingFunc)

getPendingCompactionOperations

public final Stream<Pair<String,CompactionOperation>> getPendingCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingCompactionOperations in interface TableFileSystemView

Returns:

Stream>

getPartitionPaths

public final List<StoragePath> getPartitionPaths()

getPartitionNames

public final List<String> getPartitionNames()

getPendingLogCompactionOperations

public final Stream<Pair<String,CompactionOperation>> getPendingLogCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingLogCompactionOperations in interface TableFileSystemView

Returns:

Stream>

getLatestBaseFiles

public final Stream<HoodieBaseFile> getLatestBaseFiles(String partitionStr)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files in the given partition.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFiles

public final Stream<HoodieBaseFile> getLatestBaseFiles()

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files, in the file system view.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFilesBeforeOrOn

public final Stream<HoodieBaseFile> getLatestBaseFilesBeforeOrOn(String partitionStr,
                                                                 String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getAllLatestBaseFilesBeforeOrOn

public final Map<String,Stream<HoodieBaseFile>> getAllLatestBaseFilesBeforeOrOn(String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Streams the latest version base files in all partitions with precondition that commitTime(file) before maxCommitTime.

Specified by:

getAllLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Parameters:

maxCommitTime - The max commit time to consider.

Returns:

A Map of partition path to the latest version base files before or on the commit time

getBaseFileOn

public final Option<HoodieBaseFile> getBaseFileOn(String partitionStr,
                                                  String instantTime,
                                                  String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Get the version of data file matching the instant time in the given partition.

Specified by:

getBaseFileOn in interface TableFileSystemView.BaseFileOnlyView

getLatestBaseFile

public final Option<HoodieBaseFile> getLatestBaseFile(String partitionStr,
                                                      String fileId)

Get Latest base file for a partition and file-Id.

Specified by:

getLatestBaseFile in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFilesInRange

public final Stream<HoodieBaseFile> getLatestBaseFilesInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files pass.

Specified by:

getLatestBaseFilesInRange in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

loadAllPartitions

public void loadAllPartitions()

Description copied from interface: TableFileSystemView

Load all partition and file slices into view

Specified by:

loadAllPartitions in interface TableFileSystemView

loadPartitions

public void loadPartitions(List<String> partitionPaths)

Description copied from interface: TableFileSystemView

Load all partition and file slices into view for the provided partition paths

Specified by:

loadPartitions in interface TableFileSystemView

Parameters:

partitionPaths - List of partition paths to load

getAllBaseFiles

public final Stream<HoodieBaseFile> getAllBaseFiles(String partitionStr)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Stream all the data file versions grouped by FileId for a given partition.

Specified by:

getAllBaseFiles in interface TableFileSystemView.BaseFileOnlyView

getLatestFileSlices

public final Stream<FileSlice> getLatestFileSlices(String partitionStr)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition.

Specified by:

getLatestFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesIncludingInflight

public Stream<FileSlice> getLatestFileSlicesIncludingInflight(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Get the latest file slices for a given partition including the inflight ones.

Specified by:

getLatestFileSlicesIncludingInflight in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - The partition path of interest

Returns:

Stream of latest FileSlice in the partition path.

getLatestFileSlicesStateless

public final Stream<FileSlice> getLatestFileSlicesStateless(String partitionStr)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition without caching the file group mappings.

This is useful for some table services such as compaction and clustering, these services may search around the files to clean within some ancient data partitions, if there triggers a full table service for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

Specified by:

getLatestFileSlicesStateless in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlice

public final Option<FileSlice> getLatestFileSlice(String partitionStr,
                                                  String fileId)

Get Latest File Slice for a given fileId in a given partition.

Specified by:

getLatestFileSlice in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestUnCompactedFileSlices

public final Stream<FileSlice> getLatestUnCompactedFileSlices(String partitionStr)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest uncompacted file slices in the given partition.

Specified by:

getLatestUnCompactedFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesBeforeOrOn

public final Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionStr,
                                                             String maxCommitTime,
                                                             boolean includeFileSlicesInPendingCompaction)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionStr - Partition path

maxCommitTime - Max Instant Time

includeFileSlicesInPendingCompaction - include file-slices that are in pending compaction

getAllLatestFileSlicesBeforeOrOn

public final Map<String,Stream<FileSlice>> getAllLatestFileSlicesBeforeOrOn(String maxCommitTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.

Specified by:

getAllLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

maxCommitTime - Max Instant Time

Returns:

A Map of partition path to the latest file slices before maxCommitTime.

getLatestMergedFileSlicesBeforeOrOn

public final Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionStr,
                                                                   String maxInstantTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.

Specified by:

getLatestMergedFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionStr - Partition Path

maxInstantTime - Max Instant Time

Returns:

getAllLogsMergedFileSliceBeforeOrOn

public final Stream<FileSlice> getAllLogsMergedFileSliceBeforeOrOn(String partitionStr,
                                                                   String maxInstantTime)

Stream all "merged" file-slices before on an instant time for a MERGE_ON_READ table with index that can index log files(which means it writes pure logs first).

In streaming read scenario, in order for better reading efficiency, the user can choose to skip the base files that are produced by compaction. That is to say, we allow the users to consumer only from these partitioned log files, these log files keep the record sequence just like the normal message queue.

NOTE: only local view is supported.

Parameters:

partitionStr - Partition Path

maxInstantTime - Max Instant Time

getLatestFileSliceInRange

public final Stream<FileSlice> getLatestFileSliceInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices, in the given range.

Specified by:

getLatestFileSliceInRange in interface TableFileSystemView.SliceViewWithLatestSlice

getAllFileSlices

public final Stream<FileSlice> getAllFileSlices(String partitionStr)

Description copied from interface: TableFileSystemView.SliceView

Stream all the file slices for a given partition, latest or not.

Specified by:

getAllFileSlices in interface TableFileSystemView.SliceView

getAllFileGroups

public final Stream<HoodieFileGroup> getAllFileGroups(String partitionStr)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition.

Specified by:

getAllFileGroups in interface TableFileSystemView

getAllFileGroupsStateless

public final Stream<HoodieFileGroup> getAllFileGroupsStateless(String partitionStr)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition without caching the file group mappings.

This is useful for some table services such as cleaning, the cleaning service may search around the files to clean within some ancient data partitions, if there triggers a full table cleaning for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

Specified by:

getAllFileGroupsStateless in interface TableFileSystemView

getReplacedFileGroupsBeforeOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsBeforeOrOn(String maxCommitTime,
                                                               String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before or on maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBeforeOrOn in interface TableFileSystemView

getReplacedFileGroupsBefore

public Stream<HoodieFileGroup> getReplacedFileGroupsBefore(String maxCommitTime,
                                                           String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBefore in interface TableFileSystemView

getReplacedFileGroupsAfterOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsAfterOrOn(String minCommitTime,
                                                              String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups after or on minCommitTime.

Specified by:

getReplacedFileGroupsAfterOrOn in interface TableFileSystemView

getAllReplacedFileGroups

public Stream<HoodieFileGroup> getAllReplacedFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups for given partition.

Specified by:

getAllReplacedFileGroups in interface TableFileSystemView

getFileGroupsInPendingClustering

public final Stream<Pair<HoodieFileGroupId,HoodieInstant>> getFileGroupsInPendingClustering()

Description copied from interface: TableFileSystemView

Filegroups that are in pending clustering.

Specified by:

getFileGroupsInPendingClustering in interface TableFileSystemView

isPendingCompactionScheduledForFileId

protected abstract boolean isPendingCompactionScheduledForFileId(HoodieFileGroupId fgId)

Check if there is an outstanding compaction scheduled for this file.

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending compaction, false otherwise

isPendingLogCompactionScheduledForFileId

protected abstract boolean isPendingLogCompactionScheduledForFileId(HoodieFileGroupId fgId)

Check if there is an outstanding log compaction scheduled for this file.

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending log compaction, false otherwise

isPendingClusteringScheduledForFileId

protected abstract boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId)

Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.

Parameters:

fgId - File-Group Id

Returns:

true if there is a pending clustering, false otherwise

getPendingClusteringInstant

protected abstract Option<HoodieInstant> getPendingClusteringInstant(HoodieFileGroupId fileGroupId)

Get pending clustering instant time for specified file group. Return None if file group is not in pending clustering operation.

fetchFileGroupsInPendingClustering

protected abstract Stream<Pair<HoodieFileGroupId,HoodieInstant>> fetchFileGroupsInPendingClustering()

Fetch all file groups in pending clustering.

getPendingCompactionOperationWithInstant

protected abstract Option<Pair<String,CompactionOperation>> getPendingCompactionOperationWithInstant(HoodieFileGroupId fileGroupId)

Return pending compaction operation for a file-group.

Parameters:

fileGroupId - File-Group Id

getPendingLogCompactionOperationWithInstant

protected abstract Option<Pair<String,CompactionOperation>> getPendingLogCompactionOperationWithInstant(HoodieFileGroupId fileGroupId)

Return pending Log compaction operation for a file-group.

Parameters:

fileGroupId - File-Group Id

isBootstrapBaseFilePresentForFileId

protected abstract boolean isBootstrapBaseFilePresentForFileId(HoodieFileGroupId fgId)

Check if there is a bootstrap base file present for this file.

Parameters:

fgId - File-Group Id

Returns:

true if there is associated bootstrap base-file, false otherwise

getBootstrapBaseFile

protected abstract Option<BootstrapBaseFileMapping> getBootstrapBaseFile(HoodieFileGroupId fileGroupId)

Return pending compaction operation for a file-group.

Parameters:

fileGroupId - File-Group Id

resetReplacedFileGroups

protected abstract void resetReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Track instant time for file groups replaced.

addReplacedFileGroups

protected abstract void addReplacedFileGroups(Map<HoodieFileGroupId,HoodieInstant> replacedFileGroups)

Track instant time for new file groups replaced.

removeReplacedFileIdsAtInstants

protected abstract void removeReplacedFileIdsAtInstants(Set<String> instants)

Remove file groups that are replaced in any of the specified instants.

hasReplacedFilesInPartition

protected abstract boolean hasReplacedFilesInPartition(String partitionPath)

Returns whether there are replaced files within the given partition.

getReplaceInstant

protected abstract Option<HoodieInstant> getReplaceInstant(HoodieFileGroupId fileGroupId)

Track instant time for file groups replaced.

fetchLatestBaseFiles

public Stream<HoodieBaseFile> fetchLatestBaseFiles(String partitionPath)

Default implementation for fetching latest base-files for the partition-path.

getLatestBaseFile

protected Option<HoodieBaseFile> getLatestBaseFile(HoodieFileGroup fileGroup)

fetchLatestBaseFile

protected Option<HoodieBaseFile> fetchLatestBaseFile(String partitionPath,
                                                     String fileId)

Default implementation for fetching latest base-file.

Parameters:

partitionPath - Partition path

fileId - File Id

Returns:

base File if present

fetchLatestFileSlice

public Option<FileSlice> fetchLatestFileSlice(String partitionPath,
                                              String fileId)

Default implementation for fetching file-slice.

Parameters:

partitionPath - Partition path

fileId - File Id

Returns:

File Slice if present

getLastInstant

public Option<HoodieInstant> getLastInstant()

Description copied from interface: TableFileSystemView

Last Known Instant on which the view is built.

Specified by:

getLastInstant in interface TableFileSystemView

getTimeline

public HoodieTimeline getTimeline()

Description copied from interface: TableFileSystemView

Timeline corresponding to the view.

Specified by:

getTimeline in interface TableFileSystemView

sync

public void sync()

Syncs the file system view from storage to memory. Performs complete reset of file-system view. Subsequent partition view calls will load file slices against the latest timeline.

NOTE: The logic MUST BE guarded by the write lock.

Specified by:

sync in interface SyncableFileSystemView

getVisibleCommitsAndCompactionTimeline

public HoodieTimeline getVisibleCommitsAndCompactionTimeline()

Return Only Commits and Compaction timeline for building file-groups.

Returns:

HoodieTimeline