org.apache.beam.sdk.io.hadoop.format

Class HadoopFormatIO.HadoopInputFormatBoundedSource<K,V>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>

org.apache.beam.sdk.io.hadoop.format.HadoopFormatIO.HadoopInputFormatBoundedSource<K,V>

Type Parameters:

K - Type of keys to be read.

V - Type of values to be read.

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

HadoopFormatIO

public static class HadoopFormatIO.HadoopInputFormatBoundedSource<K,V>
extends org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>
implements java.io.Serializable

Bounded source implementation for HadoopFormatIO.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

org.apache.beam.sdk.io.BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

org.apache.beam.sdk.io.Source.Reader<T>

Constructor Summary

Constructors

Modifier

Constructor and Description

protected

HadoopInputFormatBoundedSource(org.apache.beam.sdk.io.hadoop.SerializableConfiguration conf, org.apache.beam.sdk.coders.Coder<K> keyCoder, org.apache.beam.sdk.coders.Coder<V> valueCoder, @Nullable org.apache.beam.sdk.transforms.SimpleFunction<?,K> keyTranslationFunction, @Nullable org.apache.beam.sdk.transforms.SimpleFunction<?,V> valueTranslationFunction, HadoopFormatIO.SerializableSplit inputSplit, boolean skipKeyClone, boolean skipValueClone)

Method Summary

Modifier and Type	Method and Description
protected void	createInputFormatInstance() Creates instance of InputFormat class.
org.apache.beam.sdk.io.BoundedSource.BoundedReader<org.apache.beam.sdk.values.KV<K,V>>	createReader(org.apache.beam.sdk.options.PipelineOptions options)
org.apache.beam.sdk.io.hadoop.SerializableConfiguration	getConfiguration()
long	getEstimatedSizeBytes(org.apache.beam.sdk.options.PipelineOptions po)
org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<K,V>>	getOutputCoder()
void	populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
java.util.List<org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>>	split(long desiredBundleSizeBytes, org.apache.beam.sdk.options.PipelineOptions options)
void	validate()

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopInputFormatBoundedSource

protected HadoopInputFormatBoundedSource(org.apache.beam.sdk.io.hadoop.SerializableConfiguration conf,
                                         org.apache.beam.sdk.coders.Coder<K> keyCoder,
                                         org.apache.beam.sdk.coders.Coder<V> valueCoder,
                                         @Nullable org.apache.beam.sdk.transforms.SimpleFunction<?,K> keyTranslationFunction,
                                         @Nullable org.apache.beam.sdk.transforms.SimpleFunction<?,V> valueTranslationFunction,
                                         HadoopFormatIO.SerializableSplit inputSplit,
                                         boolean skipKeyClone,
                                         boolean skipValueClone)

Method Detail

getConfiguration

public org.apache.beam.sdk.io.hadoop.SerializableConfiguration getConfiguration()

validate

public void validate()

Overrides:

validate in class org.apache.beam.sdk.io.Source<org.apache.beam.sdk.values.KV<K,V>>

populateDisplayData

public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)

Specified by:

populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData

Overrides:

populateDisplayData in class org.apache.beam.sdk.io.Source<org.apache.beam.sdk.values.KV<K,V>>

split

public java.util.List<org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>> split(long desiredBundleSizeBytes,
                                                                                                      org.apache.beam.sdk.options.PipelineOptions options)
                                                                                               throws java.lang.Exception

Specified by:

split in class org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>

Throws:

java.lang.Exception

getEstimatedSizeBytes

public long getEstimatedSizeBytes(org.apache.beam.sdk.options.PipelineOptions po)
                           throws java.lang.Exception

Specified by:

getEstimatedSizeBytes in class org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>

Throws:

java.lang.Exception

createInputFormatInstance

protected void createInputFormatInstance()
                                  throws java.io.IOException

Creates instance of InputFormat class. The InputFormat class name is specified in the Hadoop configuration.

Throws:

java.io.IOException

getOutputCoder

public org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<K,V>> getOutputCoder()

Overrides:

getOutputCoder in class org.apache.beam.sdk.io.Source<org.apache.beam.sdk.values.KV<K,V>>

createReader

public org.apache.beam.sdk.io.BoundedSource.BoundedReader<org.apache.beam.sdk.values.KV<K,V>> createReader(org.apache.beam.sdk.options.PipelineOptions options)
                                                                                                    throws java.io.IOException

Specified by:

createReader in class org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<K,V>>

Throws:

java.io.IOException