org.apache.beam.sdk.io

Class AvroIO.Read

java.lang.Object

org.apache.beam.sdk.io.AvroIO.Read

Enclosing class:

AvroIO

public static class AvroIO.Read
extends Object

A root PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a PCollection containing the decoding of each record.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroIO.Read.Bound<T> A PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a bounded PCollection containing the decoding of each record.

Method Summary

Modifier and Type	Method and Description
static AvroIO.Read.Bound<GenericRecord>	from(String filepattern) Returns a PTransform that reads from the file(s) with the given name or pattern.
static AvroIO.Read.Bound<GenericRecord>	withoutValidation() Returns a PTransform that reads Avro file(s) that has GCS path validation on pipeline creation disabled.
static <T> AvroIO.Read.Bound<T>	withSchema(Class<T> type) Returns a PTransform that reads Avro file(s) containing records whose type is the specified Avro-generated class.
static AvroIO.Read.Bound<GenericRecord>	withSchema(Schema schema) Returns a PTransform that reads Avro file(s) containing records of the specified schema.
static AvroIO.Read.Bound<GenericRecord>	withSchema(String schema) Returns a PTransform that reads Avro file(s) containing records of the specified schema in a JSON-encoded string form.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

from

public static AvroIO.Read.Bound<GenericRecord> from(String filepattern)

Returns a PTransform that reads from the file(s) with the given name or pattern. This can be a local filename or filename pattern (if running locally), or a Google Cloud Storage filename or filename pattern of the form "gs://<bucket>/<filepath>" (if running locally or via the Google Cloud Dataflow service). Standard Java Filesystem glob patterns ("*", "?", "[..]") are supported.

withSchema

public static <T> AvroIO.Read.Bound<T> withSchema(Class<T> type)

Returns a PTransform that reads Avro file(s) containing records whose type is the specified Avro-generated class.

Type Parameters:

T - the type of the decoded elements, and the elements of the resulting PCollection

withSchema

public static AvroIO.Read.Bound<GenericRecord> withSchema(Schema schema)

Returns a PTransform that reads Avro file(s) containing records of the specified schema.

withSchema

public static AvroIO.Read.Bound<GenericRecord> withSchema(String schema)

Returns a PTransform that reads Avro file(s) containing records of the specified schema in a JSON-encoded string form.

withoutValidation

public static AvroIO.Read.Bound<GenericRecord> withoutValidation()

Returns a PTransform that reads Avro file(s) that has GCS path validation on pipeline creation disabled.

This can be useful in the case where the GCS input location does not exist at the pipeline creation time, but is expected to be available at execution time.