org.apache.hudi.common.table.timeline

Class LSMTimeline

java.lang.Object

org.apache.hudi.common.table.timeline.LSMTimeline

public class LSMTimeline
extends Object

Represents the LSM Timeline for the Hoodie table.

After several instants are accumulated as a batch on the active timeline, they would be flushed as a parquet file into the LSM timeline. In general the timeline is comprised with parquet files with LSM style file layout. Each new operation to the timeline yields a new snapshot version. Theoretically, there could be multiple snapshot versions on the timeline.

The LSM Timeline Layout

   t111, t112 ... t120 ... ->
     \              /
        \        /
            |
            V
   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
                                  \                    /
                                     \              /
                                            |
                                            V
                                    t11_t100_1.parquet                L1

      manifest_1, manifest_2, ... manifest_12
                                      |
                                      V
                                  _version_
 

The LSM Tree Compaction

Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer. We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1, so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we apply concurrent read.

The benchmark shows 1000 instants reading cost about 10 ms.

The Archiver & Reader Snapshot Isolation

In order to make snapshot isolation of the LSM timeline write/read, we add two kinds of metadata files for the LSM tree version management:

1. Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;
2. Version file: A version file is generated right after a new manifest file is formed.

The Reader Workflow

• read the latest version;
• read the manifest file for valid file handles;
• read the data files, probably do a data skipping with the parquet file name max min timestamp.

The Legacy Files Cleaning and Read Retention

Only triggers file cleaning after a valid compaction.

Clean Strategy

Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span, which is far longer that the LSM timeline loading time.

Instants TTL

The timeline reader only reads instants of last limited days. We will by default skip the instants from LSM timeline that are generated long time ago.

Field Summary

Fields

Modifier and Type	Field and Description
static int	LSM_TIMELINE_INSTANT_VERSION_1

Constructor Summary

Constructors

Constructor and Description
LSMTimeline()

Method Summary

Modifier and Type	Method and Description
static List<Integer>	allSnapshotVersions(HoodieTableMetaClient metaClient) Returns all the valid snapshot versions.
static int	getFileLayer(String fileName) Parse the layer number from the file name.
static StoragePath	getManifestFilePath(HoodieTableMetaClient metaClient, int snapshotVersion) Returns the full manifest file path with given version number.
static StoragePathFilter	getManifestFilePathFilter() Returns a path filter for the manifest files.
static int	getManifestVersion(String fileName) Parse the snapshot version from the manifest file name.
static String	getMaxInstantTime(String fileName) Parse the maximum instant time from the file name.
static String	getMinInstantTime(String fileName) Parse the minimum instant time from the file name.
static org.apache.avro.Schema	getReadSchema(HoodieArchivedTimeline.LoadMode loadMode)
static StoragePath	getVersionFilePath(HoodieTableMetaClient metaClient) Returns the full version file path with given version number.
static boolean	isFileFromLayer(String fileName, int layer) Returns whether a file belongs to the specified layer layer within the LSM layout.
static boolean	isFileInRange(HoodieArchivedTimeline.TimeRangeFilter filter, String fileName) Returns whether the given file is located in the filter.
static HoodieLSMTimelineManifest	latestSnapshotManifest(HoodieTableMetaClient metaClient) Returns the latest snapshot metadata files.
static HoodieLSMTimelineManifest	latestSnapshotManifest(HoodieTableMetaClient metaClient, int latestVersion) Reads the file list from the manifest file for the latest snapshot.
static int	latestSnapshotVersion(HoodieTableMetaClient metaClient) Returns the latest snapshot version.
static List<StoragePathInfo>	listAllManifestFiles(HoodieTableMetaClient metaClient) List all the parquet manifest files.
static List<StoragePathInfo>	listAllMetaFiles(HoodieTableMetaClient metaClient) List all the parquet metadata files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LSM_TIMELINE_INSTANT_VERSION_1

public static final int LSM_TIMELINE_INSTANT_VERSION_1

See Also:

Constant Field Values

Constructor Detail

LSMTimeline

public LSMTimeline()

Method Detail

getReadSchema

public static org.apache.avro.Schema getReadSchema(HoodieArchivedTimeline.LoadMode loadMode)

isFileInRange

public static boolean isFileInRange(HoodieArchivedTimeline.TimeRangeFilter filter,
                                    String fileName)

Returns whether the given file is located in the filter.

latestSnapshotVersion

public static int latestSnapshotVersion(HoodieTableMetaClient metaClient)
                                 throws IOException

Returns the latest snapshot version.

Throws:

IOException

allSnapshotVersions

public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient)
                                         throws IOException

Returns all the valid snapshot versions.

Throws:

IOException

latestSnapshotManifest

public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient)
                                                        throws IOException

Returns the latest snapshot metadata files.

Throws:

IOException

latestSnapshotManifest

public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient,
                                                               int latestVersion)

Reads the file list from the manifest file for the latest snapshot.

getManifestFilePath

public static StoragePath getManifestFilePath(HoodieTableMetaClient metaClient,
                                              int snapshotVersion)

Returns the full manifest file path with given version number.

getVersionFilePath

public static StoragePath getVersionFilePath(HoodieTableMetaClient metaClient)

Returns the full version file path with given version number.

listAllManifestFiles

public static List<StoragePathInfo> listAllManifestFiles(HoodieTableMetaClient metaClient)
                                                  throws IOException

List all the parquet manifest files.

Throws:

IOException

listAllMetaFiles

public static List<StoragePathInfo> listAllMetaFiles(HoodieTableMetaClient metaClient)
                                              throws IOException

List all the parquet metadata files.

Throws:

IOException

getManifestVersion

public static int getManifestVersion(String fileName)

Parse the snapshot version from the manifest file name.

getFileLayer

public static int getFileLayer(String fileName)

Parse the layer number from the file name.

getMinInstantTime

public static String getMinInstantTime(String fileName)

Parse the minimum instant time from the file name.

getMaxInstantTime

public static String getMaxInstantTime(String fileName)

Parse the maximum instant time from the file name.

isFileFromLayer

public static boolean isFileFromLayer(String fileName,
                                      int layer)

Returns whether a file belongs to the specified layer layer within the LSM layout.

getManifestFilePathFilter

public static StoragePathFilter getManifestFilePathFilter()

Returns a path filter for the manifest files.