org.apache.flink.api.java.record.io

Class CsvInputFormat

java.lang.Object

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<OT>

org.apache.flink.api.common.io.GenericCsvInputFormat<org.apache.flink.types.Record>

org.apache.flink.api.java.record.io.CsvInputFormat

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.InputFormat<org.apache.flink.types.Record,org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>

public class CsvInputFormat
extends org.apache.flink.api.common.io.GenericCsvInputFormat<org.apache.flink.types.Record>

Input format to parse text files and generate Records. The input file is structured by record delimiters and field delimiters (CSV files are common). Record delimiter separate records from each other ('\n' is common). Field delimiters separate fields within a record. Record and field delimiters must be configured using the InputFormat Configuration. The number of fields to parse must be configured as well. For each field a data type must be specified using the FIELD_TYPE_PARAMETER_PREFIX config key. The position within the text record can be configured for each field using the TEXT_POSITION_PARAMETER_PREFIX config key. Either all text positions must be configured or none. If none is configured, the index of the config key is used. The position of a value within the Record is the index of the config key.

See Also:

Configuration, Record, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	CsvInputFormat.AbstractConfigBuilder<T> An abstract builder used to set parameters to the input format's configuration in a fluent way.
static class	CsvInputFormat.ConfigBuilder A builder used to set parameters to the input format's configuration in a fluent way.

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

commentCount, commentPrefix, fieldIncluded, invalidLineCount

Fields inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

RECORD_DELIMITER

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
CsvInputFormat()
CsvInputFormat(char fieldDelimiter)
CsvInputFormat(char fieldDelimiter, Class<? extends org.apache.flink.types.Value>... fields)
CsvInputFormat(Class<? extends org.apache.flink.types.Value>... fields)

Method Summary

Modifier and Type	Method and Description
void	configure(org.apache.flink.configuration.Configuration config)
static CsvInputFormat.ConfigBuilder	configureRecordFormat(FileDataSource target) Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.
void	open(org.apache.flink.core.fs.FileInputSplit split)
org.apache.flink.types.Record	readRecord(org.apache.flink.types.Record reuse, byte[] bytes, int offset, int numBytes)
void	setFields(int[] sourceFieldIndices, Class<? extends org.apache.flink.types.Value>[] fieldTypes)
void	setFieldTypes(Class<? extends org.apache.flink.types.Value>... fieldTypes)
void	setFieldTypesArray(Class<? extends org.apache.flink.types.Value>[] fieldTypes)

Methods inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

checkAndCoSort, checkForMonotonousOrder, close, enableQuotedStringParsing, getCommentPrefix, getFieldDelimiter, getFieldParsers, getGenericFieldTypes, getNumberOfFieldsTotal, getNumberOfNonNullFields, isLenient, isSkippingFirstLineAsHeader, parseRecord, setCommentPrefix, setCommentPrefix, setCommentPrefix, setCommentPrefix, setCommentPrefix, setFieldDelimiter, setFieldDelimiter, setFieldDelimiter, setFieldsGeneric, setFieldsGeneric, setFieldTypesGeneric, setLenient, setSkipFirstLineAsHeader, skipFields

Methods inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

configureDelimitedFormat, getBufferSize, getDelimiter, getLineLengthLimit, getNumLineSamples, getStatistics, loadGloablConfigParams, nextRecord, reachedEnd, readLine, setBufferSize, setDelimiter, setDelimiter, setDelimiter, setLineLengthLimit, setNumLineSamples

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, configureFileFormat, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

CsvInputFormat

public CsvInputFormat()

CsvInputFormat

public CsvInputFormat(char fieldDelimiter)

CsvInputFormat

public CsvInputFormat(Class<? extends org.apache.flink.types.Value>... fields)

CsvInputFormat

public CsvInputFormat(char fieldDelimiter,
                      Class<? extends org.apache.flink.types.Value>... fields)

Method Detail

setFieldTypesArray

public void setFieldTypesArray(Class<? extends org.apache.flink.types.Value>[] fieldTypes)

setFieldTypes

public void setFieldTypes(Class<? extends org.apache.flink.types.Value>... fieldTypes)

setFields

public void setFields(int[] sourceFieldIndices,
                      Class<? extends org.apache.flink.types.Value>[] fieldTypes)

configure

public void configure(org.apache.flink.configuration.Configuration config)

Specified by:

configure in interface org.apache.flink.api.common.io.InputFormat<org.apache.flink.types.Record,org.apache.flink.core.fs.FileInputSplit>

Overrides:

configure in class org.apache.flink.api.common.io.DelimitedInputFormat<org.apache.flink.types.Record>

open

public void open(org.apache.flink.core.fs.FileInputSplit split)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.io.InputFormat<org.apache.flink.types.Record,org.apache.flink.core.fs.FileInputSplit>

Overrides:

open in class org.apache.flink.api.common.io.GenericCsvInputFormat<org.apache.flink.types.Record>

Throws:

IOException

readRecord

public org.apache.flink.types.Record readRecord(org.apache.flink.types.Record reuse,
                                                byte[] bytes,
                                                int offset,
                                                int numBytes)
                                         throws org.apache.flink.api.common.io.ParseException

Specified by:

readRecord in class org.apache.flink.api.common.io.DelimitedInputFormat<org.apache.flink.types.Record>

Throws:

org.apache.flink.api.common.io.ParseException

configureRecordFormat

public static CsvInputFormat.ConfigBuilder configureRecordFormat(FileDataSource target)

Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.

Returns:

A config builder for setting parameters.