org.apache.hudi.io

Class HoodieMergeHandle<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.io.HoodieIOHandle<T,I,K,O>

org.apache.hudi.io.HoodieWriteHandle<T,I,K,O>

org.apache.hudi.io.HoodieMergeHandle<T,I,K,O>

Direct Known Subclasses:

HoodieConcatHandle, HoodieSortedMergeHandle

@NotThreadSafe
public class HoodieMergeHandle<T extends HoodieRecordPayload,I,K,O>
extends HoodieWriteHandle<T,I,K,O>

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieFileWriter<org.apache.avro.generic.IndexedRecord>	fileWriter
protected long	insertRecordsWritten
protected Option<BaseKeyGenerator>	keyGeneratorOpt
protected Map<String,HoodieRecord<T>>	keyToNewRecords
protected org.apache.hadoop.fs.Path	newFilePath
protected org.apache.hadoop.fs.Path	oldFilePath
protected long	recordsDeleted
protected long	recordsWritten
protected long	updatedRecordsWritten
protected boolean	useWriterSchemaForCompaction
protected Set<String>	writtenRecordKeys

Fields inherited from class org.apache.hudi.io.HoodieWriteHandle

fileId, IGNORE_RECORD, partitionPath, schemaOnReadEnabled, tableSchema, tableSchemaWithMetaFields, taskContextSupplier, timer, writeSchema, writeSchemaWithMetaFields, writeStatus, writeToken

Fields inherited from class org.apache.hudi.io.HoodieIOHandle

config, fs, hoodieTable, instantTime

Constructor Summary

Constructors

Constructor and Description
HoodieMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId, TaskContextSupplier taskContextSupplier, HoodieBaseFile baseFile, Option<BaseKeyGenerator> keyGeneratorOpt)
HoodieMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId, TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt)
HoodieMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T,I,K,O> hoodieTable, Map<String,HoodieRecord<T>> keyToNewRecords, String partitionPath, String fileId, HoodieBaseFile dataFileToBeMerged, TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt) Called by compactor code path.

Method Summary

Modifier and Type	Method and Description
HoodieBaseFile	baseFileForMerge()
List<WriteStatus>	close()
IOType	getIOType()
org.apache.hadoop.fs.Path	getOldFilePath()
org.apache.avro.Schema	getWriterSchema()
org.apache.avro.Schema	getWriterSchemaWithMetaFields()
protected void	init(String fileId, Iterator<HoodieRecord<T>> newRecordsItr) Load the new incoming records in a map and return partitionPath.
protected void	initializeIncomingRecordsMap() Initialize a spillable map for incoming records.
protected void	makeOldAndNewFilePaths(String partitionPath, String oldFileName, String newFileName)
void	performMergeDataValidationCheck(WriteStatus writeStatus)
protected void	setWriteStatusPath()
void	write(org.apache.avro.generic.GenericRecord oldRecord) Go through an old record.
protected void	writeIncomingRecords()
protected void	writeInsertRecord(HoodieRecord<T> hoodieRecord)
protected boolean	writeRecord(HoodieRecord<T> hoodieRecord, Option<org.apache.avro.generic.IndexedRecord> indexedRecord)
protected boolean	writeRecord(HoodieRecord<T> hoodieRecord, Option<org.apache.avro.generic.IndexedRecord> indexedRecord, boolean isDelete)
protected void	writeToFile(HoodieKey key, org.apache.avro.generic.GenericRecord avroRecord, boolean shouldPreserveRecordMetadata)

Methods inherited from class org.apache.hudi.io.HoodieWriteHandle

canWrite, createMarkerFile, createNewFileWriter, getAttemptId, getFileSystem, getPartitionId, getPartitionPath, getStageId, makeNewFilePath, makeNewPath, rewriteRecord, rewriteRecordWithMetadata, write, write, writeStatuses

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

keyToNewRecords

protected Map<String,HoodieRecord<T extends HoodieRecordPayload>> keyToNewRecords

writtenRecordKeys

protected Set<String> writtenRecordKeys

fileWriter

protected HoodieFileWriter<org.apache.avro.generic.IndexedRecord> fileWriter

newFilePath

protected org.apache.hadoop.fs.Path newFilePath

oldFilePath

protected org.apache.hadoop.fs.Path oldFilePath

recordsWritten

protected long recordsWritten

recordsDeleted

protected long recordsDeleted

updatedRecordsWritten

protected long updatedRecordsWritten

insertRecordsWritten

protected long insertRecordsWritten

useWriterSchemaForCompaction

protected boolean useWriterSchemaForCompaction

keyGeneratorOpt

protected Option<BaseKeyGenerator> keyGeneratorOpt

Constructor Detail

HoodieMergeHandle

public HoodieMergeHandle(HoodieWriteConfig config,
                         String instantTime,
                         HoodieTable<T,I,K,O> hoodieTable,
                         Iterator<HoodieRecord<T>> recordItr,
                         String partitionPath,
                         String fileId,
                         TaskContextSupplier taskContextSupplier,
                         Option<BaseKeyGenerator> keyGeneratorOpt)

HoodieMergeHandle

public HoodieMergeHandle(HoodieWriteConfig config,
                         String instantTime,
                         HoodieTable<T,I,K,O> hoodieTable,
                         Iterator<HoodieRecord<T>> recordItr,
                         String partitionPath,
                         String fileId,
                         TaskContextSupplier taskContextSupplier,
                         HoodieBaseFile baseFile,
                         Option<BaseKeyGenerator> keyGeneratorOpt)

HoodieMergeHandle

public HoodieMergeHandle(HoodieWriteConfig config,
                         String instantTime,
                         HoodieTable<T,I,K,O> hoodieTable,
                         Map<String,HoodieRecord<T>> keyToNewRecords,
                         String partitionPath,
                         String fileId,
                         HoodieBaseFile dataFileToBeMerged,
                         TaskContextSupplier taskContextSupplier,
                         Option<BaseKeyGenerator> keyGeneratorOpt)

Called by compactor code path.

Method Detail

getWriterSchemaWithMetaFields

public org.apache.avro.Schema getWriterSchemaWithMetaFields()

Overrides:

getWriterSchemaWithMetaFields in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

getWriterSchema

public org.apache.avro.Schema getWriterSchema()

setWriteStatusPath

protected void setWriteStatusPath()

makeOldAndNewFilePaths

protected void makeOldAndNewFilePaths(String partitionPath,
                                      String oldFileName,
                                      String newFileName)

initializeIncomingRecordsMap

protected void initializeIncomingRecordsMap()

Initialize a spillable map for incoming records.

init

protected void init(String fileId,
                    Iterator<HoodieRecord<T>> newRecordsItr)

Load the new incoming records in a map and return partitionPath.

writeInsertRecord

protected void writeInsertRecord(HoodieRecord<T> hoodieRecord)
                          throws IOException

Throws:

IOException

writeRecord

protected boolean writeRecord(HoodieRecord<T> hoodieRecord,
                              Option<org.apache.avro.generic.IndexedRecord> indexedRecord)

writeRecord

protected boolean writeRecord(HoodieRecord<T> hoodieRecord,
                              Option<org.apache.avro.generic.IndexedRecord> indexedRecord,
                              boolean isDelete)

write

public void write(org.apache.avro.generic.GenericRecord oldRecord)

Go through an old record. Here if we detect a newer version shows up, we write the new one to the file.

writeToFile

protected void writeToFile(HoodieKey key,
                           org.apache.avro.generic.GenericRecord avroRecord,
                           boolean shouldPreserveRecordMetadata)
                    throws IOException

Throws:

IOException

writeIncomingRecords

protected void writeIncomingRecords()
                             throws IOException

Throws:

IOException

close

public List<WriteStatus> close()

Specified by:

close in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

performMergeDataValidationCheck

public void performMergeDataValidationCheck(WriteStatus writeStatus)

getOldFilePath

public org.apache.hadoop.fs.Path getOldFilePath()

getIOType

public IOType getIOType()

Specified by:

getIOType in class HoodieWriteHandle<T extends HoodieRecordPayload,I,K,O>

baseFileForMerge

public HoodieBaseFile baseFileForMerge()