org.apache.hudi.source.stats

Class ColumnStatsIndices

java.lang.Object

org.apache.hudi.source.stats.ColumnStatsIndices

public class ColumnStatsIndices
extends Object

Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index, providing convenient interfaces to read it, transpose, etc.

Method Summary

Modifier and Type	Method and Description
static List<org.apache.flink.table.data.RowData>	readColumnStatsIndex(String basePath, HoodieMetadataConfig metadataConfig, String[] targetColumns)
static Pair<List<org.apache.flink.table.data.RowData>,String[]>	transposeColumnStatsIndex(List<org.apache.flink.table.data.RowData> colStats, String[] queryColumns, org.apache.flink.table.types.logical.RowType tableSchema) Transposes and converts the raw table format of the Column Stats Index representation, where each row/record corresponds to individual (column, file) pair, into the table format where each row corresponds to single file with statistic for individual columns collated w/in such row:

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

readColumnStatsIndex

public static List<org.apache.flink.table.data.RowData> readColumnStatsIndex(String basePath,
                                                                             HoodieMetadataConfig metadataConfig,
                                                                             String[] targetColumns)

transposeColumnStatsIndex

public static Pair<List<org.apache.flink.table.data.RowData>,String[]> transposeColumnStatsIndex(List<org.apache.flink.table.data.RowData> colStats,
                                                                                                 String[] queryColumns,
                                                                                                 org.apache.flink.table.types.logical.RowType tableSchema)

Transposes and converts the raw table format of the Column Stats Index representation, where each row/record corresponds to individual (column, file) pair, into the table format where each row corresponds to single file with statistic for individual columns collated w/in such row:

Metadata Table Column Stats Index format:

  +---------------------------+------------+------------+------------+-------------+
  |        fileName           | columnName |  minValue  |  maxValue  |  num_nulls  |
  +---------------------------+------------+------------+------------+-------------+
  | one_base_file.parquet     |          A |          1 |         10 |           0 |
  | another_base_file.parquet |          A |        -10 |          0 |           5 |
  +---------------------------+------------+------------+------------+-------------+
 

Returned table format

  +---------------------------+------------+------------+-------------+
  |          file             | A_minValue | A_maxValue | A_nullCount |
  +---------------------------+------------+------------+-------------+
  | one_base_file.parquet     |          1 |         10 |           0 |
  | another_base_file.parquet |        -10 |          0 |           5 |
  +---------------------------+------------+------------+-------------+
 

NOTE: Column Stats Index might potentially contain statistics for many columns (if not all), while query at hand might only be referencing a handful of those. As such, we collect all the column references from the filtering expressions, and only transpose records corresponding to the columns referenced in those

Parameters:

colStats - RowData list bearing raw Column Stats Index table

queryColumns - target columns to be included into the final table

tableSchema - schema of the source data table

Returns:

reshaped table according to the format outlined above