org.apache.hudi.table.format.cow

Class CopyOnWriteInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,org.apache.flink.core.fs.FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

org.apache.hudi.table.format.cow.CopyOnWriteInputFormat

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.InputFormat<org.apache.flink.table.data.RowData,org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>

public class CopyOnWriteInputFormat
extends org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

An implementation of FileInputFormat to read RowData records from Parquet files.

Note: Reference Flink release 1.11.2 org.apache.flink.formats.parquet.ParquetFileSystemFormatFactory.ParquetInputFormat to support TIMESTAMP_MILLIS.

Note: Override the createInputSplits(int) method from parent to rewrite the logic creating the FileSystem, use FSUtils.getFs(java.lang.String, org.apache.hadoop.conf.Configuration) to get a plugin filesystem.

See Also:

ParquetSplitReaderUtil, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
CopyOnWriteInputFormat(org.apache.flink.core.fs.Path[] paths, String[] fullFieldNames, org.apache.flink.table.types.DataType[] fullFieldTypes, int[] selectedFields, String partDefaultName, long limit, org.apache.hadoop.conf.Configuration conf, boolean utcTimestamp)

Method Summary

Modifier and Type	Method and Description
boolean	acceptFile(org.apache.hadoop.fs.FileStatus fileStatus) A simple hook to filter files and directories from the input.
void	close()
org.apache.flink.core.fs.FileInputSplit[]	createInputSplits(int minNumSplits)
org.apache.flink.table.data.RowData	nextRecord(org.apache.flink.table.data.RowData reuse)
void	open(org.apache.flink.core.fs.FileInputSplit fileSplit)
boolean	reachedEnd()
void	setFilesFilter(org.apache.flink.api.common.io.FilePathFilter filesFilter)
boolean	supportsMultiPaths()

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, configure, decorateInputStream, extractFileExtension, getFilePath, getFilePaths, getFileStats, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNestedFileEnumeration, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, getStatistics, registerInflaterInputStreamFactory, setFilePath, setFilePath, setFilePaths, setFilePaths, setMinSplitSize, setNestedFileEnumeration, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

CopyOnWriteInputFormat

public CopyOnWriteInputFormat(org.apache.flink.core.fs.Path[] paths,
                              String[] fullFieldNames,
                              org.apache.flink.table.types.DataType[] fullFieldTypes,
                              int[] selectedFields,
                              String partDefaultName,
                              long limit,
                              org.apache.hadoop.conf.Configuration conf,
                              boolean utcTimestamp)

Method Detail

open

public void open(org.apache.flink.core.fs.FileInputSplit fileSplit)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.io.InputFormat<org.apache.flink.table.data.RowData,org.apache.flink.core.fs.FileInputSplit>

Overrides:

open in class org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

Throws:

IOException

createInputSplits

public org.apache.flink.core.fs.FileInputSplit[] createInputSplits(int minNumSplits)
                                                            throws IOException

Specified by:

createInputSplits in interface org.apache.flink.api.common.io.InputFormat<org.apache.flink.table.data.RowData,org.apache.flink.core.fs.FileInputSplit>

Specified by:

createInputSplits in interface org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>

Overrides:

createInputSplits in class org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

Throws:

IOException

supportsMultiPaths

public boolean supportsMultiPaths()

Overrides:

supportsMultiPaths in class org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

reachedEnd

public boolean reachedEnd()
                   throws IOException

Throws:

IOException

nextRecord

public org.apache.flink.table.data.RowData nextRecord(org.apache.flink.table.data.RowData reuse)

close

public void close()
           throws IOException

Specified by:

close in interface org.apache.flink.api.common.io.InputFormat<org.apache.flink.table.data.RowData,org.apache.flink.core.fs.FileInputSplit>

Overrides:

close in class org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

Throws:

IOException

setFilesFilter

public void setFilesFilter(org.apache.flink.api.common.io.FilePathFilter filesFilter)

Overrides:

setFilesFilter in class org.apache.flink.api.common.io.FileInputFormat<org.apache.flink.table.data.RowData>

acceptFile

public boolean acceptFile(org.apache.hadoop.fs.FileStatus fileStatus)

A simple hook to filter files and directories from the input. The method may be overridden. Hadoop's FileInputFormat has a similar mechanism and applies the same filters by default.

Parameters:

fileStatus - The file status to check.

Returns:

true, if the given file or directory is accepted