| Constructor and Description |
|---|
BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
HoodieTableMetaClient metaClient,
TypedProperties configProperties,
HoodieTableQueryType queryType,
List<StoragePath> queryPaths,
Option<String> specifiedQueryInstant,
boolean shouldIncludePendingCommits,
boolean shouldValidateInstant,
BaseHoodieTableFileIndex.FileStatusCache fileStatusCache,
boolean shouldListLazily,
Option<String> beginInstantTime,
Option<String> endInstantTime) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
BootstrapIndex.metaClient |
protected HoodieTableMetaClient |
BootstrapIndex.IndexReader.metaClient |
protected HoodieTableMetaClient |
BootstrapIndex.IndexWriter.metaClient |
| Modifier and Type | Method and Description |
|---|---|
static BootstrapIndex |
BootstrapIndex.getBootstrapIndex(HoodieTableMetaClient metaClient) |
| Constructor and Description |
|---|
BootstrapIndex(HoodieTableMetaClient metaClient) |
IndexReader(HoodieTableMetaClient metaClient) |
IndexWriter(HoodieTableMetaClient metaClient) |
NoOpBootstrapIndex(HoodieTableMetaClient metaClient) |
| Constructor and Description |
|---|
HFileBootstrapIndex(HoodieTableMetaClient metaClient) |
HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
TableSchemaResolver.metaClient |
| Modifier and Type | Method and Description |
|---|---|
HoodieTableMetaClient |
HoodieTableMetaClient.Builder.build() |
HoodieTableMetaClient |
HoodieTableMetaClient.PropertyBuilder.initTable(StorageConfiguration<?> configuration,
StoragePath basePath) |
HoodieTableMetaClient |
HoodieTableMetaClient.PropertyBuilder.initTable(StorageConfiguration<?> configuration,
String basePath)
Init Table with the properties build by this builder.
|
static HoodieTableMetaClient |
HoodieTableMetaClient.initTableAndGetMetaClient(StorageConfiguration<?> storageConf,
StoragePath basePath,
Properties props) |
static HoodieTableMetaClient |
HoodieTableMetaClient.initTableAndGetMetaClient(StorageConfiguration<?> storageConf,
String basePath,
Properties props)
Helper method to initialize a given path as a hoodie table with configs passed in as Properties.
|
static HoodieTableMetaClient |
HoodieTableMetaClient.reload(HoodieTableMetaClient oldMetaClient) |
| Modifier and Type | Method and Description |
|---|---|
void |
HoodieTableConfig.clearMetadataPartitions(HoodieTableMetaClient metaClient)
|
HoodieTableMetaClient.PropertyBuilder |
HoodieTableMetaClient.PropertyBuilder.fromMetaClient(HoodieTableMetaClient metaClient) |
static HoodieTableMetaClient |
HoodieTableMetaClient.reload(HoodieTableMetaClient oldMetaClient) |
void |
HoodieTableConfig.setMetadataPartitionsInflight(HoodieTableMetaClient metaClient,
List<String> partitionPaths)
Enables the specified metadata table partition as inflight.
|
void |
HoodieTableConfig.setMetadataPartitionsInflight(HoodieTableMetaClient metaClient,
MetadataPartitionType... partitionTypes) |
void |
HoodieTableConfig.setMetadataPartitionState(HoodieTableMetaClient metaClient,
String partitionPath,
boolean enabled)
Enables or disables the specified metadata table partition.
|
| Constructor and Description |
|---|
TableSchemaResolver(HoodieTableMetaClient metaClient) |
| Constructor and Description |
|---|
HoodieCDCExtractor(HoodieTableMetaClient metaClient,
InstantRange range) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
AbstractHoodieLogRecordReader.hoodieTableMetaClient |
protected HoodieTableMetaClient |
BaseHoodieLogRecordReader.hoodieTableMetaClient |
protected HoodieTableMetaClient |
HoodieMetadataMergedLogRecordScanner.Builder.hoodieTableMetaClient |
protected HoodieTableMetaClient |
HoodieMergedLogRecordScanner.Builder.hoodieTableMetaClient |
| Modifier and Type | Method and Description |
|---|---|
AbstractHoodieLogRecordReader.Builder |
AbstractHoodieLogRecordReader.Builder.withTableMetaClient(HoodieTableMetaClient hoodieTableMetaClient) |
HoodieMetadataMergedLogRecordScanner.Builder |
HoodieMetadataMergedLogRecordScanner.Builder.withTableMetaClient(HoodieTableMetaClient hoodieTableMetaClient) |
HoodieMergedLogRecordScanner.Builder |
HoodieMergedLogRecordScanner.Builder.withTableMetaClient(HoodieTableMetaClient hoodieTableMetaClient) |
HoodieUnMergedLogRecordScanner.Builder |
HoodieUnMergedLogRecordScanner.Builder.withTableMetaClient(HoodieTableMetaClient hoodieTableMetaClient) |
| Constructor and Description |
|---|
AbstractHoodieLogRecordReader(HoodieStorage storage,
String basePath,
List<String> logFilePaths,
org.apache.avro.Schema readerSchema,
String latestInstantTime,
boolean reverseReader,
int bufferSize,
Option<InstantRange> instantRange,
boolean withOperationField,
boolean forceFullScan,
Option<String> partitionNameOverride,
InternalSchema internalSchema,
Option<String> keyFieldOverride,
boolean enableOptimizedLogBlocksScan,
HoodieRecordMerger recordMerger,
Option<HoodieTableMetaClient> hoodieTableMetaClientOption) |
BaseHoodieMergedLogRecordScanner(HoodieStorage storage,
String basePath,
List<String> logFilePaths,
org.apache.avro.Schema readerSchema,
String latestInstantTime,
Long maxMemorySizeInBytes,
boolean reverseReader,
int bufferSize,
String spillableMapBasePath,
Option<InstantRange> instantRange,
ExternalSpillableMap.DiskMapType diskMapType,
boolean isBitCaskDiskMapCompressionEnabled,
boolean withOperationField,
boolean forceFullScan,
Option<String> partitionName,
InternalSchema internalSchema,
Option<String> keyFieldOverride,
boolean enableOptimizedLogBlocksScan,
HoodieRecordMerger recordMerger,
Option<HoodieTableMetaClient> hoodieTableMetaClientOption) |
HoodieMergedLogRecordScanner(HoodieStorage storage,
String basePath,
List<String> logFilePaths,
org.apache.avro.Schema readerSchema,
String latestInstantTime,
Long maxMemorySizeInBytes,
boolean reverseReader,
int bufferSize,
String spillableMapBasePath,
Option<InstantRange> instantRange,
ExternalSpillableMap.DiskMapType diskMapType,
boolean isBitCaskDiskMapCompressionEnabled,
boolean withOperationField,
boolean forceFullScan,
Option<String> partitionName,
InternalSchema internalSchema,
Option<String> keyFieldOverride,
boolean enableOptimizedLogBlocksScan,
HoodieRecordMerger recordMerger,
Option<HoodieTableMetaClient> hoodieTableMetaClientOption) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
HoodieBaseFileGroupRecordBuffer.hoodieTableMetaClient |
| Modifier and Type | Method and Description |
|---|---|
static HoodieTimeline |
IncrementalQueryAnalyzer.filterInstantsAsPerUserConfigs(HoodieTableMetaClient metaClient,
HoodieTimeline timeline,
boolean skipCompaction,
boolean skipClustering,
boolean skipInsertOverwrite)
Filters out the unnecessary instants as per user specified configs.
|
IncrementalQueryAnalyzer.Builder |
IncrementalQueryAnalyzer.Builder.metaClient(HoodieTableMetaClient metaClient) |
| Constructor and Description |
|---|
HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
HoodieTableMetaClient hoodieTableMetaClient,
Option<String> partitionNameOverrideOpt,
Option<String[]> partitionPathFieldOpt,
HoodieRecordMerger recordMerger,
TypedProperties props) |
HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
HoodieStorage storage,
String tablePath,
String latestCommitTime,
FileSlice fileSlice,
org.apache.avro.Schema dataSchema,
org.apache.avro.Schema requestedSchema,
Option<InternalSchema> internalSchemaOpt,
HoodieTableMetaClient hoodieTableMetaClient,
TypedProperties props,
long start,
long length,
boolean shouldUseRecordPosition) |
HoodieKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
HoodieTableMetaClient hoodieTableMetaClient,
Option<String> partitionNameOverrideOpt,
Option<String[]> partitionPathFieldOpt,
HoodieRecordMerger recordMerger,
TypedProperties props) |
HoodiePositionBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
HoodieTableMetaClient hoodieTableMetaClient,
Option<String> partitionNameOverrideOpt,
Option<String[]> partitionPathFieldOpt,
HoodieRecordMerger recordMerger,
TypedProperties props) |
HoodieUnmergedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
HoodieTableMetaClient hoodieTableMetaClient,
Option<String> partitionNameOverrideOpt,
Option<String[]> partitionPathFieldOpt,
HoodieRecordMerger recordMerger,
TypedProperties props) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
HoodieActiveTimeline.metaClient |
| Modifier and Type | Method and Description |
|---|---|
static List<Integer> |
LSMTimeline.allSnapshotVersions(HoodieTableMetaClient metaClient)
Returns all the valid snapshot versions.
|
static HoodieTimeline |
TimelineUtils.concatTimeline(HoodieTimeline timeline1,
HoodieTimeline timeline2,
HoodieTableMetaClient metaClient)
Concat two timelines timeline1 and timeline2 to build a new timeline.
|
static HoodieLSMTimelineInstant |
MetadataConversionUtils.createLSMTimelineInstant(ActiveAction activeAction,
HoodieTableMetaClient metaClient) |
static HoodieArchivedMetaEntry |
MetadataConversionUtils.createMetaWrapper(HoodieInstant hoodieInstant,
HoodieTableMetaClient metaClient) |
static Map<String,Option<String>> |
TimelineUtils.getAllExtraMetadataForKey(HoodieTableMetaClient metaClient,
String extraMetadataKey)
Get extra metadata for specified key from all active commit/deltacommit instants.
|
Option<byte[]> |
ActiveAction.getCleanPlan(HoodieTableMetaClient metaClient) |
Option<byte[]> |
ActiveAction.getCommitMetadata(HoodieTableMetaClient metaClient) |
static HoodieTimeline |
TimelineUtils.getCommitsTimelineAfter(HoodieTableMetaClient metaClient,
String exclusiveStartInstantTime,
Option<String> lastMaxCompletionTime)
Returns a Hudi timeline with commits after the given instant time (exclusive).
|
Option<byte[]> |
ActiveAction.getCompactionPlan(HoodieTableMetaClient metaClient) |
static List<String> |
TimelineUtils.getDroppedPartitions(HoodieTableMetaClient metaClient,
Option<String> lastCommitTimeSynced,
Option<String> lastCommitCompletionTimeSynced)
Returns partitions that have been deleted or marked for deletion in the timeline between given commit time range.
|
static Option<String> |
TimelineUtils.getExtraMetadataFromLatest(HoodieTableMetaClient metaClient,
String extraMetadataKey)
Get extra metadata for specified key from latest commit/deltacommit/replacecommit(eg.
|
static Option<String> |
TimelineUtils.getExtraMetadataFromLatestIncludeClustering(HoodieTableMetaClient metaClient,
String extraMetadataKey)
Get extra metadata for specified key from latest commit/deltacommit/replacecommit instant including internal commits
such as clustering.
|
static Option<HoodieCommitMetadata> |
MetadataConversionUtils.getHoodieCommitMetadata(HoodieTableMetaClient metaClient,
HoodieInstant hoodieInstant) |
Option<byte[]> |
ActiveAction.getInflightCommitMetadata(HoodieTableMetaClient metaClient) |
static HoodieInstant |
HoodieTimeline.getInflightInstant(HoodieInstant instant,
HoodieTableMetaClient metaClient)
Returns the inflight instant corresponding to the instant being passed.
|
Option<byte[]> |
ActiveAction.getLogCompactionPlan(HoodieTableMetaClient metaClient) |
static StoragePath |
LSMTimeline.getManifestFilePath(HoodieTableMetaClient metaClient,
int snapshotVersion)
Returns the full manifest file path with given version number.
|
Option<byte[]> |
ActiveAction.getRequestedCommitMetadata(HoodieTableMetaClient metaClient) |
static HoodieDefaultTimeline |
TimelineUtils.getTimeline(HoodieTableMetaClient metaClient,
boolean includeArchivedTimeline) |
static StoragePath |
LSMTimeline.getVersionFilePath(HoodieTableMetaClient metaClient)
Returns the full version file path with given version number.
|
static HoodieTimeline |
TimelineUtils.handleHollowCommitIfNeeded(HoodieTimeline completedCommitTimeline,
HoodieTableMetaClient metaClient,
TimelineUtils.HollowCommitHandling handlingMode)
Handles hollow commit as per
HoodieCommonConfig.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT
and return filtered or non-filtered timeline for incremental query to run against. |
static boolean |
TimelineUtils.isClusteringCommit(HoodieTableMetaClient metaClient,
HoodieInstant instant) |
static HoodieLSMTimelineManifest |
LSMTimeline.latestSnapshotManifest(HoodieTableMetaClient metaClient)
Returns the latest snapshot metadata files.
|
static HoodieLSMTimelineManifest |
LSMTimeline.latestSnapshotManifest(HoodieTableMetaClient metaClient,
int latestVersion)
Reads the file list from the manifest file for the latest snapshot.
|
static int |
LSMTimeline.latestSnapshotVersion(HoodieTableMetaClient metaClient)
Returns the latest snapshot version.
|
static List<StoragePathInfo> |
LSMTimeline.listAllManifestFiles(HoodieTableMetaClient metaClient)
List all the parquet manifest files.
|
static List<StoragePathInfo> |
LSMTimeline.listAllMetaFiles(HoodieTableMetaClient metaClient)
List all the parquet metadata files.
|
static void |
HoodieArchivedTimeline.loadInstants(HoodieTableMetaClient metaClient,
HoodieArchivedTimeline.TimeRangeFilter filter,
HoodieArchivedTimeline.LoadMode loadMode,
Function<org.apache.avro.generic.GenericRecord,Boolean> commitsFilter,
BiConsumer<String,org.apache.avro.generic.GenericRecord> recordConsumer)
Loads the instants from the timeline.
|
static void |
TimelineUtils.validateTimestampAsOf(HoodieTableMetaClient metaClient,
String timestampAsOf)
Validate user-specified timestamp of time travel query against incomplete commit's timestamp.
|
| Constructor and Description |
|---|
CompletionTimeQueryView(HoodieTableMetaClient metaClient)
The constructor.
|
CompletionTimeQueryView(HoodieTableMetaClient metaClient,
String eagerLoadInstant)
The constructor.
|
HoodieActiveTimeline(HoodieTableMetaClient metaClient) |
HoodieActiveTimeline(HoodieTableMetaClient metaClient,
boolean applyLayoutFilter) |
HoodieActiveTimeline(HoodieTableMetaClient metaClient,
Set<String> includedExtensions) |
HoodieActiveTimeline(HoodieTableMetaClient metaClient,
Set<String> includedExtensions,
boolean applyLayoutFilters) |
HoodieArchivedTimeline(HoodieTableMetaClient metaClient)
Loads all the archived instants.
|
HoodieArchivedTimeline(HoodieTableMetaClient metaClient,
String startTs)
Loads completed instants from startTs(inclusive).
|
| Modifier and Type | Method and Description |
|---|---|
static Stream<HoodieFileGroup> |
DTOUtils.fileGroupDTOsToFileGroups(List<FileGroupDTO> dtos,
HoodieTableMetaClient metaClient) |
static HoodieTimeline |
TimelineDTO.toTimeline(TimelineDTO dto,
HoodieTableMetaClient metaClient) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
AbstractMigratorBase.metaClient |
| Constructor and Description |
|---|
AbstractMigratorBase(HoodieTableMetaClient metaClient) |
MetadataMigrator(HoodieTableMetaClient metaClient,
List<VersionMigrator<T>> migratorList) |
| Constructor and Description |
|---|
CleanMetadataMigrator(HoodieTableMetaClient metaClient) |
CleanMetadataV1MigrationHandler(HoodieTableMetaClient metaClient) |
CleanMetadataV2MigrationHandler(HoodieTableMetaClient metaClient) |
CleanPlanMigrator(HoodieTableMetaClient metaClient) |
CleanPlanV1MigrationHandler(HoodieTableMetaClient metaClient) |
CleanPlanV2MigrationHandler(HoodieTableMetaClient metaClient) |
| Constructor and Description |
|---|
CompactionPlanMigrator(HoodieTableMetaClient metaClient) |
CompactionV1MigrationHandler(HoodieTableMetaClient metaClient) |
CompactionV2MigrationHandler(HoodieTableMetaClient metaClient) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
AbstractTableFileSystemView.metaClient |
| Modifier and Type | Method and Description |
|---|---|
static HoodieTableFileSystemView |
FileSystemViewManager.createInMemoryFileSystemView(HoodieEngineContext engineContext,
HoodieTableMetaClient metaClient,
HoodieMetadataConfig metadataConfig) |
static HoodieTableFileSystemView |
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(HoodieEngineContext engineContext,
HoodieTableMetaClient metaClient,
HoodieMetadataConfig metadataConfig,
HoodieTimeline timeline) |
SyncableFileSystemView |
FileSystemViewManager.getFileSystemView(HoodieTableMetaClient metaClient)
Main API to get the file-system view for the base-path.
|
void |
HoodieTableFileSystemView.init(HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline) |
protected void |
RocksDbBasedFileSystemView.init(HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline) |
protected void |
AbstractTableFileSystemView.init(HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline)
Initialize the view.
|
void |
HoodieTableFileSystemView.init(HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline,
List<StoragePathInfo> pathInfoList)
Visible for testing
|
| Modifier and Type | Method and Description |
|---|---|
static FileSystemViewManager |
FileSystemViewManager.createViewManager(HoodieEngineContext context,
FileSystemViewStorageConfig config,
HoodieCommonConfig commonConfig,
SerializableFunctionUnchecked<HoodieTableMetaClient,HoodieTableMetadata> metadataCreator)
Main Factory method for building file-system views.
|
| Modifier and Type | Method and Description |
|---|---|
static Map<HoodieFileGroupId,HoodieInstant> |
ClusteringUtils.getAllFileGroupsInPendingClusteringPlans(HoodieTableMetaClient metaClient)
Get filegroups to pending clustering instant mapping for all pending clustering plans.
|
static Stream<Pair<HoodieInstant,HoodieClusteringPlan>> |
ClusteringUtils.getAllPendingClusteringPlans(HoodieTableMetaClient metaClient)
Get all pending clustering plans along with their instants.
|
static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>> |
CompactionUtils.getAllPendingCompactionOperations(HoodieTableMetaClient metaClient)
Get all PartitionPath + file-ids with pending Compaction operations and their target compaction instant time.
|
static List<Pair<HoodieInstant,HoodieCompactionPlan>> |
CompactionUtils.getAllPendingCompactionPlans(HoodieTableMetaClient metaClient)
Get all pending compaction plans along with their instants.
|
static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>> |
CompactionUtils.getAllPendingLogCompactionOperations(HoodieTableMetaClient metaClient)
Get all partition + file Ids with pending Log Compaction operations and their target log compaction instant time.
|
static List<Pair<HoodieInstant,HoodieCompactionPlan>> |
CompactionUtils.getAllPendingLogCompactionPlans(HoodieTableMetaClient metaClient)
Get all pending logcompaction plans along with their instants.
|
static HoodieCleanMetadata |
CleanerUtils.getCleanerMetadata(HoodieTableMetaClient metaClient,
byte[] details)
Get Latest Version of Hoodie Cleaner Metadata - Output of cleaner operation.
|
static HoodieCleanMetadata |
CleanerUtils.getCleanerMetadata(HoodieTableMetaClient metaClient,
HoodieInstant cleanInstant)
Get Latest Version of Hoodie Cleaner Metadata - Output of cleaner operation.
|
static HoodieCleanerPlan |
CleanerUtils.getCleanerPlan(HoodieTableMetaClient metaClient,
byte[] details)
Get Latest version of cleaner plan corresponding to a clean instant.
|
static HoodieCleanerPlan |
CleanerUtils.getCleanerPlan(HoodieTableMetaClient metaClient,
HoodieInstant cleanInstant)
Get Latest version of cleaner plan corresponding to a clean instant.
|
static Option<Pair<HoodieInstant,HoodieClusteringPlan>> |
ClusteringUtils.getClusteringPlan(HoodieTableMetaClient metaClient,
HoodieInstant pendingReplaceInstant)
Get Clustering plan from timeline.
|
static HoodieCompactionPlan |
CompactionUtils.getCompactionPlan(HoodieTableMetaClient metaClient,
Option<byte[]> planContent)
Util method to fetch both compaction and log compaction plan from requestedInstant.
|
static HoodieCompactionPlan |
CompactionUtils.getCompactionPlan(HoodieTableMetaClient metaClient,
String compactionInstant)
This method will serve only Compaction instants
because we use same HoodieCompactionPlan for both the operations.
|
static Option<HoodieInstant> |
ClusteringUtils.getEarliestInstantToRetainForClustering(HoodieActiveTimeline activeTimeline,
HoodieTableMetaClient metaClient)
Returns the earliest instant to retain.
|
static Pair<Option<String>,Option<String>> |
InternalSchemaCache.getInternalSchemaAndAvroSchemaForClusteringAndCompaction(HoodieTableMetaClient metaClient,
String compactionAndClusteringInstant)
Get internalSchema and avroSchema for compaction/cluster operation.
|
static InternalSchema |
InternalSchemaCache.getInternalSchemaByVersionId(long versionId,
HoodieTableMetaClient metaClient) |
static HoodieCompactionPlan |
CompactionUtils.getLogCompactionPlan(HoodieTableMetaClient metaClient,
String logCompactionInstant)
This method will serve only log compaction instants,
because we use same HoodieCompactionPlan for both the operations.
|
static List<HoodieInstant> |
ClusteringUtils.getPendingClusteringInstantTimes(HoodieTableMetaClient metaClient) |
static List<HoodieInstant> |
CompactionUtils.getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient)
Return all pending compaction instant times.
|
static InternalSchema |
InternalSchemaCache.searchSchemaAndCache(long versionID,
HoodieTableMetaClient metaClient)
Search internalSchema based on versionID.
|
| Constructor and Description |
|---|
RocksDBSchemaHelper(HoodieTableMetaClient metaClient) |
| Modifier and Type | Method and Description |
|---|---|
void |
SecondaryIndexManager.create(HoodieTableMetaClient metaClient,
String indexName,
String indexType,
boolean ignoreIfExists,
LinkedHashMap<String,Map<String,String>> columns,
Map<String,String> options)
Create a secondary index for hoodie table, two steps will be performed:
1.
|
void |
SecondaryIndexManager.drop(HoodieTableMetaClient metaClient,
String indexName,
boolean ignoreIfNotExists)
Drop a secondary index by index name
|
static Option<List<HoodieSecondaryIndex>> |
SecondaryIndexUtils.getSecondaryIndexes(HoodieTableMetaClient metaClient)
Get secondary index metadata for this table
|
void |
SecondaryIndexManager.refresh(HoodieTableMetaClient metaClient,
String indexName)
Refresh the specific secondary index
|
Option<List<HoodieSecondaryIndex>> |
SecondaryIndexManager.show(HoodieTableMetaClient metaClient)
Show secondary indexes from hoodie table
|
| Constructor and Description |
|---|
FileBasedInternalSchemaStorageManager(HoodieTableMetaClient metaClient) |
| Modifier and Type | Field and Description |
|---|---|
protected HoodieTableMetaClient |
BaseTableMetadata.dataMetaClient |
| Modifier and Type | Method and Description |
|---|---|
HoodieTableMetaClient |
HoodieBackedTableMetadata.getMetadataMetaClient() |
| Modifier and Type | Method and Description |
|---|---|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(HoodieEngineContext engineContext,
Map<String,List<String>> partitionToDeletedFiles,
Map<String,Map<String,Long>> partitionToAppendedFiles,
String instantTime,
HoodieTableMetaClient dataMetaClient,
int bloomIndexParallelism,
String bloomFilterType)
Convert added and deleted files metadata to bloom filter index records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(HoodieEngineContext engineContext,
Map<String,List<String>> partitionToDeletedFiles,
Map<String,Map<String,Long>> partitionToAppendedFiles,
HoodieTableMetaClient dataMetaClient,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert added and deleted action metadata to column stats index records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(HoodieEngineContext engineContext,
List<HoodieTableMetadataUtil.DirectoryInfo> partitionInfoList,
HoodieMetadataConfig metadataConfig,
HoodieTableMetaClient dataTableMetaClient) |
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToBloomFilterRecords(HoodieEngineContext context,
HoodieConfig hoodieConfig,
HoodieCommitMetadata commitMetadata,
String instantTime,
HoodieTableMetaClient dataMetaClient,
String bloomFilterType,
int bloomIndexParallelism)
Convert commit action metadata to bloom filter records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords(HoodieCleanMetadata cleanMetadata,
HoodieEngineContext engineContext,
HoodieTableMetaClient dataMetaClient,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert clean metadata to column stats index records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords(HoodieCommitMetadata commitMetadata,
HoodieEngineContext engineContext,
HoodieTableMetaClient dataMetaClient,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex) |
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToPartitionStatsRecords(HoodieCommitMetadata commitMetadata,
HoodieEngineContext engineContext,
HoodieTableMetaClient dataMetaClient,
HoodieMetadataConfig metadataConfig) |
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMetadataToRecords(HoodieEngineContext engineContext,
HoodieCleanMetadata cleanMetadata,
String instantTime,
HoodieTableMetaClient dataMetaClient,
List<MetadataPartitionType> enabledPartitionTypes,
int bloomIndexParallelism,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert the clean action to metadata records.
|
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMetadataToRecords(HoodieEngineContext context,
HoodieConfig hoodieConfig,
HoodieCommitMetadata commitMetadata,
String instantTime,
HoodieTableMetaClient dataMetaClient,
List<MetadataPartitionType> enabledPartitionTypes,
String bloomFilterType,
int bloomIndexParallelism,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex,
HoodieMetadataConfig metadataConfig)
Convert commit action to metadata records for the enabled partition types.
|
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMetadataToRecords(HoodieEngineContext engineContext,
HoodieTableMetaClient dataTableMetaClient,
HoodieRollbackMetadata rollbackMetadata,
String instantTime)
Convert rollback action metadata to metadata table records.
|
static String |
HoodieTableMetadataUtil.deleteMetadataTable(HoodieTableMetaClient dataMetaClient,
HoodieEngineContext context,
boolean backup)
Delete the metadata table for the dataset and backup if required.
|
static String |
HoodieTableMetadataUtil.deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient,
HoodieEngineContext context,
String partitionPath,
boolean backup)
Delete a partition within the metadata table.
|
static List<MetadataPartitionType> |
MetadataPartitionType.getEnabledPartitions(TypedProperties writeConfig,
HoodieTableMetaClient metaClient)
Returns the list of metadata partition types enabled based on the metadata config and table config.
|
static HoodieTableFileSystemView |
HoodieTableMetadataUtil.getFileSystemView(HoodieTableMetaClient metaClient)
Get metadata table file system view.
|
static boolean |
HoodieTableMetadataUtil.getMetadataPartitionsNeedingWriteStatusTracking(HoodieMetadataConfig config,
HoodieTableMetaClient metaClient)
Returns true if any enabled metadata partition in the given hoodie table requires WriteStatus to track the written records.
|
static List<FileSlice> |
HoodieTableMetadataUtil.getPartitionLatestFileSlices(HoodieTableMetaClient metaClient,
Option<HoodieTableFileSystemView> fsView,
String partition)
Get the latest file slices for a Metadata Table partition.
|
static List<FileSlice> |
HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(HoodieTableMetaClient metaClient,
Option<HoodieTableFileSystemView> fileSystemView,
String partition)
Get the latest file slices for a given partition including the inflight ones.
|
static List<FileSlice> |
HoodieTableMetadataUtil.getPartitionLatestMergedFileSlices(HoodieTableMetaClient metaClient,
HoodieTableFileSystemView fsView,
String partition)
Get the latest file slices for a Metadata Table partition.
|
static org.apache.avro.Schema |
HoodieTableMetadataUtil.getProjectedSchemaForFunctionalIndex(HoodieIndexDefinition indexDefinition,
HoodieTableMetaClient metaClient) |
Map<String,String> |
HoodieMetadataMetrics.getStats(boolean detailed,
HoodieTableMetaClient metaClient,
HoodieTableMetadata metadata,
Set<String> metadataPartitions) |
static Set<String> |
HoodieTableMetadataUtil.getValidInstantTimestamps(HoodieTableMetaClient dataMetaClient,
HoodieTableMetaClient metadataMetaClient) |
static boolean |
HoodieTableMetadataUtil.isFilesPartitionAvailable(HoodieTableMetaClient metaClient)
Returns whether the files partition of metadata table is ready for read.
|
boolean |
MetadataPartitionType.isMetadataPartitionAvailable(HoodieTableMetaClient metaClient)
Check if the metadata partition is available based on the table config.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.readRecordKeysFromFileSlices(HoodieEngineContext engineContext,
List<Pair<String,FileSlice>> partitionFileSlicePairs,
boolean forDelete,
int recordIndexMaxParallelism,
String activeModule,
HoodieTableMetaClient metaClient,
EngineType engineType)
Reads the record keys from the given file slices and returns a
HoodieData of HoodieRecord to be updated in the metadata table. |
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.readSecondaryKeysFromBaseFiles(HoodieEngineContext engineContext,
List<Pair<String,Pair<String,List<String>>>> partitionFiles,
int secondaryIndexMaxParallelism,
String activeModule,
HoodieTableMetaClient metaClient,
EngineType engineType,
HoodieIndexDefinition indexDefinition) |
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.readSecondaryKeysFromFileSlices(HoodieEngineContext engineContext,
List<Pair<String,FileSlice>> partitionFileSlicePairs,
int secondaryIndexMaxParallelism,
String activeModule,
HoodieTableMetaClient metaClient,
EngineType engineType,
HoodieIndexDefinition indexDefinition) |
void |
HoodieMetadataMetrics.updateSizeMetrics(HoodieTableMetaClient metaClient,
HoodieBackedTableMetadata metadata,
Set<String> metadataPartitions) |
HoodieMetadataLogRecordReader.Builder |
HoodieMetadataLogRecordReader.Builder.withTableMetaClient(HoodieTableMetaClient hoodieTableMetaClient) |
| Constructor and Description |
|---|
HoodieMetadataFileSystemView(HoodieEngineContext engineContext,
HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline,
HoodieMetadataConfig metadataConfig) |
HoodieMetadataFileSystemView(HoodieTableMetaClient metaClient,
HoodieTimeline visibleActiveTimeline,
HoodieTableMetadata tableMetadata) |
Copyright © 2024 The Apache Software Foundation. All rights reserved.