org.apache.hudi.io.storage

Class HoodieParquetWriter<T extends HoodieRecordPayload,R extends org.apache.avro.generic.IndexedRecord>

java.lang.Object

org.apache.parquet.hadoop.ParquetWriter<org.apache.avro.generic.IndexedRecord>

org.apache.hudi.io.storage.HoodieParquetWriter<T,R>

All Implemented Interfaces:

Closeable, AutoCloseable, HoodieFileWriter<R>

public class HoodieParquetWriter<T extends HoodieRecordPayload,R extends org.apache.avro.generic.IndexedRecord>
extends org.apache.parquet.hadoop.ParquetWriter<org.apache.avro.generic.IndexedRecord>
implements HoodieFileWriter<R>

HoodieParquetWriter extends the ParquetWriter to help limit the size of underlying file. Provides a way to check if the current file can take more records with the canWrite()

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.parquet.hadoop.ParquetWriter

org.apache.parquet.hadoop.ParquetWriter.Builder<T,SELF extends org.apache.parquet.hadoop.ParquetWriter.Builder<T,SELF>>

Field Summary

Fields inherited from class org.apache.parquet.hadoop.ParquetWriter

DEFAULT_BLOCK_SIZE, DEFAULT_COMPRESSION_CODEC_NAME, DEFAULT_IS_DICTIONARY_ENABLED, DEFAULT_IS_VALIDATING_ENABLED, DEFAULT_PAGE_SIZE, DEFAULT_WRITER_VERSION, MAX_PADDING_SIZE_DEFAULT, OBJECT_MODEL_NAME_PROP

Constructor Summary

Constructors

Constructor and Description
HoodieParquetWriter(String instantTime, org.apache.hadoop.fs.Path file, HoodieAvroParquetConfig parquetConfig, org.apache.avro.Schema schema, TaskContextSupplier taskContextSupplier, boolean populateMetaFields)

Method Summary

Modifier and Type	Method and Description
boolean	canWrite()
void	writeAvro(String key, org.apache.avro.generic.IndexedRecord object)
void	writeAvroWithMetadata(HoodieKey key, R avroRecord)

Methods inherited from class org.apache.parquet.hadoop.ParquetWriter

close, getDataSize, getFooter, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.io.storage.HoodieFileWriter

close, prepRecordWithMetadata

Constructor Detail

HoodieParquetWriter

public HoodieParquetWriter(String instantTime,
                           org.apache.hadoop.fs.Path file,
                           HoodieAvroParquetConfig parquetConfig,
                           org.apache.avro.Schema schema,
                           TaskContextSupplier taskContextSupplier,
                           boolean populateMetaFields)
                    throws IOException

Throws:

IOException

Method Detail

writeAvroWithMetadata

public void writeAvroWithMetadata(HoodieKey key,
                                  R avroRecord)
                           throws IOException

Specified by:

writeAvroWithMetadata in interface HoodieFileWriter<R extends org.apache.avro.generic.IndexedRecord>

Throws:

IOException

canWrite

public boolean canWrite()

Specified by:

canWrite in interface HoodieFileWriter<R extends org.apache.avro.generic.IndexedRecord>

writeAvro

public void writeAvro(String key,
                      org.apache.avro.generic.IndexedRecord object)
               throws IOException

Specified by:

writeAvro in interface HoodieFileWriter<R extends org.apache.avro.generic.IndexedRecord>

Throws:

IOException