org.apache.hudi.table.marker

Class DirectWriteMarkers

java.lang.Object

org.apache.hudi.table.marker.WriteMarkers

org.apache.hudi.table.marker.DirectWriteMarkers

All Implemented Interfaces:

Serializable

public class DirectWriteMarkers
extends WriteMarkers

Marker operations of directly accessing the file system to create and delete marker files. Each data file has a corresponding marker file.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.table.marker.WriteMarkers

basePath, instantTime, markerDirPath

Constructor Summary

Constructors

Constructor and Description
DirectWriteMarkers(org.apache.hadoop.fs.FileSystem fs, String basePath, String markerFolderPath, String instantTime)
DirectWriteMarkers(HoodieTable table, String instantTime)

Method Summary

Modifier and Type	Method and Description
Set<String>	allMarkerFilePaths()
Option<org.apache.hadoop.fs.Path>	create(String markerName) Creates a marker file based on the full marker name excluding the base path and instant.
protected Option<org.apache.hadoop.fs.Path>	create(String partitionPath, String dataFileName, IOType type, boolean checkIfExists) Creates a marker.
Set<String>	createdAndMergedDataPaths(HoodieEngineContext context, int parallelism)
boolean	deleteMarkerDir(HoodieEngineContext context, int parallelism) Deletes Marker directory corresponding to an instant.
boolean	doesMarkerDirExist()

Methods inherited from class org.apache.hudi.table.marker.WriteMarkers

create, createIfNotExists, getMarkerFileName, getMarkerPath, quietDeleteMarkerDir, stripMarkerSuffix

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DirectWriteMarkers

public DirectWriteMarkers(org.apache.hadoop.fs.FileSystem fs,
                          String basePath,
                          String markerFolderPath,
                          String instantTime)

DirectWriteMarkers

public DirectWriteMarkers(HoodieTable table,
                          String instantTime)

Method Detail

deleteMarkerDir

public boolean deleteMarkerDir(HoodieEngineContext context,
                               int parallelism)

Deletes Marker directory corresponding to an instant.

Specified by:

deleteMarkerDir in class WriteMarkers

Parameters:

context - HoodieEngineContext.

parallelism - parallelism for deletion.

Returns:

if successful; otherwise.

doesMarkerDirExist

public boolean doesMarkerDirExist()
                           throws IOException

Specified by:

doesMarkerDirExist in class WriteMarkers

Returns:

true if marker directory exists; false otherwise.

Throws:

IOException

createdAndMergedDataPaths

public Set<String> createdAndMergedDataPaths(HoodieEngineContext context,
                                             int parallelism)
                                      throws IOException

Specified by:

createdAndMergedDataPaths in class WriteMarkers

Parameters:

context - HoodieEngineContext instance.

parallelism - parallelism for reading the marker files in the directory.

Returns:

all the data file paths of write IO type "CREATE" and "MERGE"

Throws:

IOException

allMarkerFilePaths

public Set<String> allMarkerFilePaths()
                               throws IOException

Specified by:

allMarkerFilePaths in class WriteMarkers

Returns:

all the marker paths

Throws:

IOException

create

public Option<org.apache.hadoop.fs.Path> create(String markerName)

Creates a marker file based on the full marker name excluding the base path and instant.

Parameters:

markerName - the full marker name, e.g., "2021/08/13/file1.marker.CREATE"

Returns:

path of the marker file

create

protected Option<org.apache.hadoop.fs.Path> create(String partitionPath,
                                                   String dataFileName,
                                                   IOType type,
                                                   boolean checkIfExists)

Description copied from class: WriteMarkers

Creates a marker.

Parameters:

partitionPath - partition path in the table

dataFileName - data file name

type - write IO type

checkIfExists - whether to check if the marker already exists

Returns:

the marker path or empty option if already exists and checkIfExists is true