org.apache.hudi.common.table.view

Class RemoteHoodieTableFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView

All Implemented Interfaces:

Serializable, AutoCloseable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class RemoteHoodieTableFileSystemView
extends Object
implements SyncableFileSystemView, Serializable

A proxy for table file-system view which translates local View API calls to REST calls to remote timeline service.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALL_DATA_FILES_URL
static String	ALL_FILEGROUPS_FOR_PARTITION_STATELESS_URL
static String	ALL_FILEGROUPS_FOR_PARTITION_URL
static String	ALL_LATEST_BASE_FILES_BEFORE_ON_INSTANT_URL
static String	ALL_LATEST_SLICES_BEFORE_ON_INSTANT_URL
static String	ALL_REPLACED_FILEGROUPS_AFTER_OR_ON_URL
static String	ALL_REPLACED_FILEGROUPS_BEFORE_OR_ON_URL
static String	ALL_REPLACED_FILEGROUPS_BEFORE_URL
static String	ALL_REPLACED_FILEGROUPS_PARTITION_URL
static String	ALL_SLICES_URL
static String	BASEPATH_PARAM
static String	FILEID_PARAM
static String	INCLUDE_FILES_IN_PENDING_COMPACTION_PARAM
static String	INSTANT_PARAM
static String	INSTANTS_PARAM
static String	LAST_INSTANT_TS
static String	LAST_INSTANT_URL
static String	LAST_INSTANTS_URL
static String	LATEST_ALL_DATA_FILES_URL
static String	LATEST_DATA_FILE_ON_INSTANT_URL
static String	LATEST_DATA_FILES_BEFORE_ON_INSTANT_URL
static String	LATEST_DATA_FILES_RANGE_INSTANT_URL
static String	LATEST_PARTITION_DATA_FILE_URL
static String	LATEST_PARTITION_DATA_FILES_URL
static String	LATEST_PARTITION_SLICE_URL
static String	LATEST_PARTITION_SLICES_INFLIGHT_URL
static String	LATEST_PARTITION_SLICES_STATELESS_URL
static String	LATEST_PARTITION_SLICES_URL
static String	LATEST_PARTITION_UNCOMPACTED_SLICES_URL
static String	LATEST_SLICES_BEFORE_ON_INSTANT_URL
static String	LATEST_SLICES_MERGED_BEFORE_ON_INSTANT_URL
static String	LATEST_SLICES_RANGE_INSTANT_URL
static String	LOAD_ALL_PARTITIONS_URL
static String	LOAD_PARTITIONS_URL
static String	MAX_INSTANT_PARAM
static String	MIN_INSTANT_PARAM
static String	MULTI_VALUE_SEPARATOR
static String	PARTITION_PARAM
static String	PARTITIONS_PARAM
static String	PENDING_CLUSTERING_FILEGROUPS_URL
static String	PENDING_COMPACTION_OPS_URL
static String	PENDING_LOG_COMPACTION_OPS_URL
static String	REFRESH_OFF
static String	REFRESH_TABLE_URL
static String	TIMELINE_HASH
static String	TIMELINE_URL

Constructor Summary

Constructors

Constructor and Description
RemoteHoodieTableFileSystemView(HoodieTableMetaClient metaClient, FileSystemViewStorageConfig viewConf)
RemoteHoodieTableFileSystemView(String server, int port, HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
void	close() Allow View to release resources and close.
Stream<HoodieBaseFile>	getAllBaseFiles(String partitionPath) Stream all the data file versions grouped by FileId for a given partition.
Stream<HoodieFileGroup>	getAllFileGroups(String partitionPath) Stream all the file groups for a given partition.
Stream<HoodieFileGroup>	getAllFileGroupsStateless(String partitionPath) Stream all the file groups for a given partition without caching the file group mappings.
Stream<FileSlice>	getAllFileSlices(String partitionPath) Stream all the file slices for a given partition, latest or not.
Map<String,Stream<HoodieBaseFile>>	getAllLatestBaseFilesBeforeOrOn(String maxCommitTime) Streams the latest version base files in all partitions with precondition that commitTime(file) before maxCommitTime.
Map<String,Stream<FileSlice>>	getAllLatestFileSlicesBeforeOrOn(String maxCommitTime) Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieFileGroup>	getAllReplacedFileGroups(String partitionPath) Stream all the replaced file groups for given partition.
Option<HoodieBaseFile>	getBaseFileOn(String partitionPath, String instantTime, String fileId) Get the version of data file matching the instant time in the given partition.
Stream<Pair<HoodieFileGroupId,HoodieInstant>>	getFileGroupsInPendingClustering() Filegroups that are in pending clustering.
Option<HoodieInstant>	getLastInstant() Last Known Instant on which the view is built.
Option<HoodieBaseFile>	getLatestBaseFile(String partitionPath, String fileId) Get Latest data file for a partition and file-Id.
Stream<HoodieBaseFile>	getLatestBaseFiles() Stream all the latest data files, in the file system view.
Stream<HoodieBaseFile>	getLatestBaseFiles(String partitionPath) Stream all the latest data files in the given partition.
Stream<HoodieBaseFile>	getLatestBaseFilesBeforeOrOn(String partitionPath, String maxCommitTime) Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieBaseFile>	getLatestBaseFilesInRange(List<String> commitsToReturn) Stream all the latest data files pass.
Option<FileSlice>	getLatestFileSlice(String partitionPath, String fileId) Get Latest File Slice for a given fileId in a given partition.
Stream<FileSlice>	getLatestFileSliceInRange(List<String> commitsToReturn) Stream all the latest file slices, in the given range.
Stream<FileSlice>	getLatestFileSlices(String partitionPath) Stream all the latest file slices in the given partition.
Stream<FileSlice>	getLatestFileSlicesBeforeOrOn(String partitionPath, String maxCommitTime, boolean includeFileSlicesInPendingCompaction) Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getLatestFileSlicesIncludingInflight(String partitionPath) Get the latest file slices for a given partition including the inflight ones.
Stream<FileSlice>	getLatestFileSlicesStateless(String partitionPath) Stream all the latest file slices in the given partition without caching the file group mappings.
Stream<FileSlice>	getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime) Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.
Stream<FileSlice>	getLatestUnCompactedFileSlices(String partitionPath) Stream all the latest uncompacted file slices in the given partition.
Stream<Pair<String,CompactionOperation>>	getPendingCompactionOperations() Return Pending Compaction Operations.
Stream<Pair<String,CompactionOperation>>	getPendingLogCompactionOperations() Return Pending Compaction Operations.
Stream<HoodieFileGroup>	getReplacedFileGroupsAfterOrOn(String minCommitTime, String partitionPath) Stream all the replaced file groups after or on minCommitTime.
Stream<HoodieFileGroup>	getReplacedFileGroupsBefore(String maxCommitTime, String partitionPath) Stream all the replaced file groups before maxCommitTime for given partition.
Stream<HoodieFileGroup>	getReplacedFileGroupsBeforeOrOn(String maxCommitTime, String partitionPath) Stream all the replaced file groups before or on maxCommitTime for given partition.
HoodieTimeline	getTimeline() Timeline corresponding to the view.
void	loadAllPartitions() Load all partition and file slices into view
void	loadPartitions(List<String> partitionPaths) Load all partition and file slices into view for the provided partition paths
boolean	refresh()
void	reset() Reset View so that they can be refreshed.
void	sync() Read the latest timeline and refresh the file-system view to match the current state of the file-system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LATEST_PARTITION_SLICES_URL

public static final String LATEST_PARTITION_SLICES_URL

LATEST_PARTITION_SLICES_INFLIGHT_URL

public static final String LATEST_PARTITION_SLICES_INFLIGHT_URL

LATEST_PARTITION_SLICES_STATELESS_URL

public static final String LATEST_PARTITION_SLICES_STATELESS_URL

LATEST_PARTITION_SLICE_URL

public static final String LATEST_PARTITION_SLICE_URL

LATEST_PARTITION_UNCOMPACTED_SLICES_URL

public static final String LATEST_PARTITION_UNCOMPACTED_SLICES_URL

ALL_SLICES_URL

public static final String ALL_SLICES_URL

LATEST_SLICES_MERGED_BEFORE_ON_INSTANT_URL

public static final String LATEST_SLICES_MERGED_BEFORE_ON_INSTANT_URL

LATEST_SLICES_RANGE_INSTANT_URL

public static final String LATEST_SLICES_RANGE_INSTANT_URL

LATEST_SLICES_BEFORE_ON_INSTANT_URL

public static final String LATEST_SLICES_BEFORE_ON_INSTANT_URL

ALL_LATEST_SLICES_BEFORE_ON_INSTANT_URL

public static final String ALL_LATEST_SLICES_BEFORE_ON_INSTANT_URL

PENDING_COMPACTION_OPS_URL

public static final String PENDING_COMPACTION_OPS_URL

PENDING_LOG_COMPACTION_OPS_URL

public static final String PENDING_LOG_COMPACTION_OPS_URL

LATEST_PARTITION_DATA_FILES_URL

public static final String LATEST_PARTITION_DATA_FILES_URL

LATEST_PARTITION_DATA_FILE_URL

public static final String LATEST_PARTITION_DATA_FILE_URL

ALL_DATA_FILES_URL

public static final String ALL_DATA_FILES_URL

LATEST_ALL_DATA_FILES_URL

public static final String LATEST_ALL_DATA_FILES_URL

LATEST_DATA_FILE_ON_INSTANT_URL

public static final String LATEST_DATA_FILE_ON_INSTANT_URL

LATEST_DATA_FILES_RANGE_INSTANT_URL

public static final String LATEST_DATA_FILES_RANGE_INSTANT_URL

LATEST_DATA_FILES_BEFORE_ON_INSTANT_URL

public static final String LATEST_DATA_FILES_BEFORE_ON_INSTANT_URL

ALL_LATEST_BASE_FILES_BEFORE_ON_INSTANT_URL

public static final String ALL_LATEST_BASE_FILES_BEFORE_ON_INSTANT_URL

ALL_FILEGROUPS_FOR_PARTITION_URL

public static final String ALL_FILEGROUPS_FOR_PARTITION_URL

ALL_FILEGROUPS_FOR_PARTITION_STATELESS_URL

public static final String ALL_FILEGROUPS_FOR_PARTITION_STATELESS_URL

ALL_REPLACED_FILEGROUPS_BEFORE_OR_ON_URL

public static final String ALL_REPLACED_FILEGROUPS_BEFORE_OR_ON_URL

ALL_REPLACED_FILEGROUPS_BEFORE_URL

public static final String ALL_REPLACED_FILEGROUPS_BEFORE_URL

ALL_REPLACED_FILEGROUPS_AFTER_OR_ON_URL

public static final String ALL_REPLACED_FILEGROUPS_AFTER_OR_ON_URL

ALL_REPLACED_FILEGROUPS_PARTITION_URL

public static final String ALL_REPLACED_FILEGROUPS_PARTITION_URL

PENDING_CLUSTERING_FILEGROUPS_URL

public static final String PENDING_CLUSTERING_FILEGROUPS_URL

LAST_INSTANT_URL

public static final String LAST_INSTANT_URL

LAST_INSTANTS_URL

public static final String LAST_INSTANTS_URL

TIMELINE_URL

public static final String TIMELINE_URL

REFRESH_TABLE_URL

public static final String REFRESH_TABLE_URL

LOAD_ALL_PARTITIONS_URL

public static final String LOAD_ALL_PARTITIONS_URL

LOAD_PARTITIONS_URL

public static final String LOAD_PARTITIONS_URL

PARTITION_PARAM

public static final String PARTITION_PARAM

See Also:

Constant Field Values

PARTITIONS_PARAM

public static final String PARTITIONS_PARAM

See Also:

Constant Field Values

BASEPATH_PARAM

public static final String BASEPATH_PARAM

See Also:

Constant Field Values

INSTANT_PARAM

public static final String INSTANT_PARAM

See Also:

Constant Field Values

MAX_INSTANT_PARAM

public static final String MAX_INSTANT_PARAM

See Also:

Constant Field Values

MIN_INSTANT_PARAM

public static final String MIN_INSTANT_PARAM

See Also:

Constant Field Values

INSTANTS_PARAM

public static final String INSTANTS_PARAM

See Also:

Constant Field Values

FILEID_PARAM

public static final String FILEID_PARAM

See Also:

Constant Field Values

LAST_INSTANT_TS

public static final String LAST_INSTANT_TS

See Also:

Constant Field Values

TIMELINE_HASH

public static final String TIMELINE_HASH

See Also:

Constant Field Values

REFRESH_OFF

public static final String REFRESH_OFF

See Also:

Constant Field Values

INCLUDE_FILES_IN_PENDING_COMPACTION_PARAM

public static final String INCLUDE_FILES_IN_PENDING_COMPACTION_PARAM

See Also:

Constant Field Values

MULTI_VALUE_SEPARATOR

public static final String MULTI_VALUE_SEPARATOR

See Also:

Constant Field Values

Constructor Detail

RemoteHoodieTableFileSystemView

public RemoteHoodieTableFileSystemView(String server,
                                       int port,
                                       HoodieTableMetaClient metaClient)

RemoteHoodieTableFileSystemView

public RemoteHoodieTableFileSystemView(HoodieTableMetaClient metaClient,
                                       FileSystemViewStorageConfig viewConf)

Method Detail

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files in the given partition.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles()

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files, in the file system view.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFilesBeforeOrOn

public Stream<HoodieBaseFile> getLatestBaseFilesBeforeOrOn(String partitionPath,
                                                           String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getAllLatestBaseFilesBeforeOrOn

public Map<String,Stream<HoodieBaseFile>> getAllLatestBaseFilesBeforeOrOn(String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Streams the latest version base files in all partitions with precondition that commitTime(file) before maxCommitTime.

Specified by:

getAllLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Parameters:

maxCommitTime - The max commit time to consider.

Returns:

A Map of partition path to the latest version base files before or on the commit time

getBaseFileOn

public Option<HoodieBaseFile> getBaseFileOn(String partitionPath,
                                            String instantTime,
                                            String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Get the version of data file matching the instant time in the given partition.

Specified by:

getBaseFileOn in interface TableFileSystemView.BaseFileOnlyView

getLatestBaseFilesInRange

public Stream<HoodieBaseFile> getLatestBaseFilesInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files pass.

Specified by:

getLatestBaseFilesInRange in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getAllBaseFiles

public Stream<HoodieBaseFile> getAllBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Stream all the data file versions grouped by FileId for a given partition.

Specified by:

getAllBaseFiles in interface TableFileSystemView.BaseFileOnlyView

getLatestBaseFile

public Option<HoodieBaseFile> getLatestBaseFile(String partitionPath,
                                                String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Get Latest data file for a partition and file-Id.

Specified by:

getLatestBaseFile in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestFileSlices

public Stream<FileSlice> getLatestFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition.

Specified by:

getLatestFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesIncludingInflight

public Stream<FileSlice> getLatestFileSlicesIncludingInflight(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Get the latest file slices for a given partition including the inflight ones.

Specified by:

getLatestFileSlicesIncludingInflight in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - The partition path of interest

Returns:

Stream of latest FileSlice in the partition path.

getLatestFileSlicesStateless

public Stream<FileSlice> getLatestFileSlicesStateless(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition without caching the file group mappings.

This is useful for some table services such as compaction and clustering, these services may search around the files to clean within some ancient data partitions, if there triggers a full table service for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

Specified by:

getLatestFileSlicesStateless in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlice

public Option<FileSlice> getLatestFileSlice(String partitionPath,
                                            String fileId)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Get Latest File Slice for a given fileId in a given partition.

Specified by:

getLatestFileSlice in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestUnCompactedFileSlices

public Stream<FileSlice> getLatestUnCompactedFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest uncompacted file slices in the given partition.

Specified by:

getLatestUnCompactedFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath,
                                                       String maxCommitTime,
                                                       boolean includeFileSlicesInPendingCompaction)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition path

maxCommitTime - Max Instant Time

includeFileSlicesInPendingCompaction - include file-slices that are in pending compaction

getAllLatestFileSlicesBeforeOrOn

public Map<String,Stream<FileSlice>> getAllLatestFileSlicesBeforeOrOn(String maxCommitTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.

Specified by:

getAllLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

maxCommitTime - Max Instant Time

Returns:

A Map of partition path to the latest file slices before maxCommitTime.

getLatestMergedFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath,
                                                             String maxInstantTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.

Specified by:

getLatestMergedFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition Path

maxInstantTime - Max Instant Time

Returns:

getLatestFileSliceInRange

public Stream<FileSlice> getLatestFileSliceInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices, in the given range.

Specified by:

getLatestFileSliceInRange in interface TableFileSystemView.SliceViewWithLatestSlice

getAllFileSlices

public Stream<FileSlice> getAllFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceView

Stream all the file slices for a given partition, latest or not.

Specified by:

getAllFileSlices in interface TableFileSystemView.SliceView

getAllFileGroups

public Stream<HoodieFileGroup> getAllFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition.

Specified by:

getAllFileGroups in interface TableFileSystemView

getAllFileGroupsStateless

public Stream<HoodieFileGroup> getAllFileGroupsStateless(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition without caching the file group mappings.

This is useful for some table services such as cleaning, the cleaning service may search around the files to clean within some ancient data partitions, if there triggers a full table cleaning for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

Specified by:

getAllFileGroupsStateless in interface TableFileSystemView

getReplacedFileGroupsBeforeOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsBeforeOrOn(String maxCommitTime,
                                                               String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before or on maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBeforeOrOn in interface TableFileSystemView

getReplacedFileGroupsBefore

public Stream<HoodieFileGroup> getReplacedFileGroupsBefore(String maxCommitTime,
                                                           String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBefore in interface TableFileSystemView

getReplacedFileGroupsAfterOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsAfterOrOn(String minCommitTime,
                                                              String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups after or on minCommitTime.

Specified by:

getReplacedFileGroupsAfterOrOn in interface TableFileSystemView

getAllReplacedFileGroups

public Stream<HoodieFileGroup> getAllReplacedFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups for given partition.

Specified by:

getAllReplacedFileGroups in interface TableFileSystemView

refresh

public boolean refresh()

loadAllPartitions

public void loadAllPartitions()

Description copied from interface: TableFileSystemView

Load all partition and file slices into view

Specified by:

loadAllPartitions in interface TableFileSystemView

loadPartitions

public void loadPartitions(List<String> partitionPaths)

Description copied from interface: TableFileSystemView

Load all partition and file slices into view for the provided partition paths

Specified by:

loadPartitions in interface TableFileSystemView

Parameters:

partitionPaths - List of partition paths to load

getPendingCompactionOperations

public Stream<Pair<String,CompactionOperation>> getPendingCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingCompactionOperations in interface TableFileSystemView

Returns:

Stream>

getPendingLogCompactionOperations

public Stream<Pair<String,CompactionOperation>> getPendingLogCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingLogCompactionOperations in interface TableFileSystemView

Returns:

Stream>

getFileGroupsInPendingClustering

public Stream<Pair<HoodieFileGroupId,HoodieInstant>> getFileGroupsInPendingClustering()

Description copied from interface: TableFileSystemView

Filegroups that are in pending clustering.

Specified by:

getFileGroupsInPendingClustering in interface TableFileSystemView

getLastInstant

public Option<HoodieInstant> getLastInstant()

Description copied from interface: TableFileSystemView

Last Known Instant on which the view is built.

Specified by:

getLastInstant in interface TableFileSystemView

getTimeline

public HoodieTimeline getTimeline()

Description copied from interface: TableFileSystemView

Timeline corresponding to the view.

Specified by:

getTimeline in interface TableFileSystemView

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface SyncableFileSystemView

reset

public void reset()

Description copied from interface: SyncableFileSystemView

Reset View so that they can be refreshed.

Specified by:

reset in interface SyncableFileSystemView

sync

public void sync()

Description copied from interface: SyncableFileSystemView

Read the latest timeline and refresh the file-system view to match the current state of the file-system. The refresh can either be done incrementally (from reading file-slices in metadata files) or from scratch by resetting view storage.

Specified by:

sync in interface SyncableFileSystemView