org.apache.hudi.table.format.mor

Class MergeOnReadInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<org.apache.flink.table.data.RowData,MergeOnReadInputSplit>

org.apache.hudi.table.format.mor.MergeOnReadInputFormat

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.InputFormat<org.apache.flink.table.data.RowData,MergeOnReadInputSplit>, org.apache.flink.core.io.InputSplitSource<MergeOnReadInputSplit>

Direct Known Subclasses:

CdcInputFormat

public class MergeOnReadInputFormat
extends org.apache.flink.api.common.io.RichInputFormat<org.apache.flink.table.data.RowData,MergeOnReadInputSplit>

The base InputFormat class to read from Hoodie data + log files.

Use ParquetRecordReader to read files instead of FSDataInputStream, overrides createInputSplits(int) and close() to change the behaviors.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MergeOnReadInputFormat.Builder Builder for MergeOnReadInputFormat.
protected static class	MergeOnReadInputFormat.LogFileOnlyIterator
protected static class	MergeOnReadInputFormat.MergeIterator

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.configuration.Configuration	conf
protected org.apache.hadoop.conf.Configuration	hadoopConf
protected MergeOnReadTableState	tableState

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MergeOnReadInputFormat(org.apache.flink.configuration.Configuration conf, MergeOnReadTableState tableState, List<org.apache.flink.table.types.DataType> fieldTypes, String defaultPartName, List<ExpressionPredicates.Predicate> predicates, long limit, boolean emitDelete, InternalSchemaManager internalSchemaManager)

Method Summary

Modifier and Type	Method and Description
static MergeOnReadInputFormat.Builder	builder() Returns the builder for MergeOnReadInputFormat.
void	close()
void	configure(org.apache.flink.configuration.Configuration configuration)
MergeOnReadInputSplit[]	createInputSplits(int minNumSplits)
protected ClosableIterator<org.apache.flink.table.data.RowData>	getBaseFileIterator(String path)
protected ClosableIterator<org.apache.flink.table.data.RowData>	getBaseFileIteratorWithMetadata(String path)
protected ClosableIterator<org.apache.flink.table.data.RowData>	getFullLogFileIterator(MergeOnReadInputSplit split)
org.apache.flink.core.io.InputSplitAssigner	getInputSplitAssigner(MergeOnReadInputSplit[] mergeOnReadInputSplits)
org.apache.flink.api.common.io.statistics.BaseStatistics	getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics baseStatistics)
protected ClosableIterator<org.apache.flink.table.data.RowData>	initIterator(MergeOnReadInputSplit split)
boolean	isClosed()
void	isEmitDelete(boolean emitDelete)
org.apache.flink.table.data.RowData	nextRecord(org.apache.flink.table.data.RowData o)
void	open(MergeOnReadInputSplit split)
boolean	reachedEnd()

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

conf

protected final org.apache.flink.configuration.Configuration conf

hadoopConf

protected transient org.apache.hadoop.conf.Configuration hadoopConf

tableState

protected final MergeOnReadTableState tableState

Constructor Detail

MergeOnReadInputFormat

protected MergeOnReadInputFormat(org.apache.flink.configuration.Configuration conf,
                                 MergeOnReadTableState tableState,
                                 List<org.apache.flink.table.types.DataType> fieldTypes,
                                 String defaultPartName,
                                 List<ExpressionPredicates.Predicate> predicates,
                                 long limit,
                                 boolean emitDelete,
                                 InternalSchemaManager internalSchemaManager)

Method Detail

builder

public static MergeOnReadInputFormat.Builder builder()

Returns the builder for MergeOnReadInputFormat.

open

public void open(MergeOnReadInputSplit split)
          throws IOException

Throws:

IOException

initIterator

protected ClosableIterator<org.apache.flink.table.data.RowData> initIterator(MergeOnReadInputSplit split)
                                                                      throws IOException

Throws:

IOException

configure

public void configure(org.apache.flink.configuration.Configuration configuration)

getStatistics

public org.apache.flink.api.common.io.statistics.BaseStatistics getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics baseStatistics)

createInputSplits

public MergeOnReadInputSplit[] createInputSplits(int minNumSplits)

getInputSplitAssigner

public org.apache.flink.core.io.InputSplitAssigner getInputSplitAssigner(MergeOnReadInputSplit[] mergeOnReadInputSplits)

reachedEnd

public boolean reachedEnd()
                   throws IOException

Throws:

IOException

nextRecord

public org.apache.flink.table.data.RowData nextRecord(org.apache.flink.table.data.RowData o)

close

public void close()
           throws IOException

Throws:

IOException

isClosed

public boolean isClosed()

getBaseFileIteratorWithMetadata

protected ClosableIterator<org.apache.flink.table.data.RowData> getBaseFileIteratorWithMetadata(String path)

getBaseFileIterator

protected ClosableIterator<org.apache.flink.table.data.RowData> getBaseFileIterator(String path)
                                                                             throws IOException

Throws:

IOException

getFullLogFileIterator

protected ClosableIterator<org.apache.flink.table.data.RowData> getFullLogFileIterator(MergeOnReadInputSplit split)

isEmitDelete

@VisibleForTesting
public void isEmitDelete(boolean emitDelete)