org.apache.flink.api.java.record.io

Class CsvInputFormat

java.lang.Object

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<OT>

org.apache.flink.api.common.io.GenericCsvInputFormat<Record>

org.apache.flink.api.java.record.io.CsvInputFormat

All Implemented Interfaces:

Serializable, InputFormat<Record,FileInputSplit>, InputSplitSource<FileInputSplit>

public class CsvInputFormat
extends GenericCsvInputFormat<Record>

Input format to parse text files and generate Records. The input file is structured by record delimiters and field delimiters (CSV files are common). Record delimiter separate records from each other ('\n' is common). Field delimiters separate fields within a record. Record and field delimiters must be configured using the InputFormat Configuration. The number of fields to parse must be configured as well. For each field a data type must be specified using the FIELD_TYPE_PARAMETER_PREFIX config key. The position within the text record can be configured for each field using the TEXT_POSITION_PARAMETER_PREFIX config key. Either all text positions must be configured or none. If none is configured, the index of the config key is used. The position of a value within the Record is the index of the config key.

See Also:

Configuration, Record, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	CsvInputFormat.AbstractConfigBuilder<T> An abstract builder used to set parameters to the input format's configuration in a fluent way.
static class	CsvInputFormat.ConfigBuilder A builder used to set parameters to the input format's configuration in a fluent way.

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

FileInputFormat.FileBaseStatistics, FileInputFormat.InputSplitOpenThread

Field Summary

Fields inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

RECORD_DELIMITER

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

DEFLATE_SUFFIX, enumerateNestedFiles, filePath, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
CsvInputFormat()
CsvInputFormat(char fieldDelimiter)
CsvInputFormat(char fieldDelimiter, Class<? extends Value>... fields)
CsvInputFormat(Class<? extends Value>... fields)

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Configuration config)
static CsvInputFormat.ConfigBuilder	configureRecordFormat(FileDataSource target) Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.
void	open(FileInputSplit split)
Record	readRecord(Record reuse, byte[] bytes, int offset, int numBytes)
void	setFields(int[] sourceFieldIndices, Class<? extends Value>[] fieldTypes)
void	setFieldTypes(Class<? extends Value>... fieldTypes)
void	setFieldTypesArray(Class<? extends Value>[] fieldTypes)

Methods inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

getFieldDelimiter, getFieldParsers, getGenericFieldTypes, getNumberOfFieldsTotal, getNumberOfNonNullFields, isLenient, isSkippingFirstLineAsHeader, parseRecord, setFieldDelimiter, setFieldsGeneric, setFieldsGeneric, setFieldTypesGeneric, setLenient, setSkipFirstLineAsHeader, skipFields

Methods inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

close, configureDelimitedFormat, getBufferSize, getDelimiter, getLineLengthLimit, getNumLineSamples, getStatistics, loadGloablConfigParams, nextRecord, reachedEnd, readLine, setBufferSize, setDelimiter, setDelimiter, setDelimiter, setDelimiter, setDelimiter, setLineLengthLimit, setNumLineSamples

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, configureFileFormat, createInputSplits, getFilePath, getFileStats, getInputSplitAssigner, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

CsvInputFormat

public CsvInputFormat()

CsvInputFormat

public CsvInputFormat(char fieldDelimiter)

CsvInputFormat

public CsvInputFormat(Class<? extends Value>... fields)

CsvInputFormat

public CsvInputFormat(char fieldDelimiter,
              Class<? extends Value>... fields)

Method Detail

setFieldTypesArray

public void setFieldTypesArray(Class<? extends Value>[] fieldTypes)

setFieldTypes

public void setFieldTypes(Class<? extends Value>... fieldTypes)

setFields

public void setFields(int[] sourceFieldIndices,
             Class<? extends Value>[] fieldTypes)

configure

public void configure(Configuration config)

Specified by:

configure in interface InputFormat<Record,FileInputSplit>

Overrides:

configure in class DelimitedInputFormat<Record>

open

public void open(FileInputSplit split)
          throws IOException

Specified by:

open in interface InputFormat<Record,FileInputSplit>

Overrides:

open in class GenericCsvInputFormat<Record>

Throws:

IOException

readRecord

public Record readRecord(Record reuse,
                byte[] bytes,
                int offset,
                int numBytes)
                  throws ParseException

Specified by:

readRecord in class DelimitedInputFormat<Record>

Throws:

ParseException

configureRecordFormat

public static CsvInputFormat.ConfigBuilder configureRecordFormat(FileDataSource target)

Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.

Returns:

A config builder for setting parameters.