org.apache.nifi.processors.hive

Class ConvertAvroToORC

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.hive.ConvertAvroToORC

All Implemented Interfaces:

ConfigurableComponent, Processor

@SideEffectFree
 @SupportsBatching
 @Tags(value={"avro","orc","hive","convert"})
 @InputRequirement(value=INPUT_REQUIRED)
 @CapabilityDescription(value="Converts an Avro record into ORC file format. This processor provides a direct mapping of an Avro record to an ORC record, such that the resulting ORC file will have the same hierarchical structure as the Avro document. If an incoming FlowFile contains a stream of multiple Avro records, the resultant FlowFile will contain a ORC file containing all of the Avro records.  If an incoming FlowFile does not contain any records, an empty ORC file is the output. NOTE: Many Avro datatypes (collections, primitives, and unions of primitives, e.g.) can be converted to ORC, but unions of collections and other complex datatypes may not be able to be converted to ORC.")
 @WritesAttribute(attribute="mime.type",description="Sets the mime type to application/octet-stream") @WritesAttribute(attribute="filename",description="Sets the filename to the existing filename with the extension replaced by / added to by .orc") @WritesAttribute(attribute="record.count",description="Sets the number of records in the ORC file.") @WritesAttribute(attribute="hive.ddl",description="Creates a partial Hive DDL statement for creating a table in Hive from this ORC file. This can be used in ReplaceText for setting the content to the DDL. To make it valid DDL, add \"LOCATION \'<path_to_orc_file_in_hdfs>\'\", where the path is the directory that contains this ORC file on HDFS. For example, ConvertAvroToORC can send flow files to a PutHDFS processor to send the file to HDFS, then to a ReplaceText to set the content to this DDL (plus the LOCATION clause as described), then to PutHiveQL processor to create the table if it doesn\'t exist.")
public class ConvertAvroToORC
extends AbstractProcessor

The ConvertAvroToORC processor takes an Avro-formatted flow file as input and converts it into ORC format.

Field Summary

Fields

Modifier and Type	Field and Description
static PropertyDescriptor	BUFFER_SIZE
static PropertyDescriptor	COMPRESSION_TYPE
static String	HIVE_DDL_ATTRIBUTE
static PropertyDescriptor	HIVE_TABLE_NAME
static PropertyDescriptor	ORC_CONFIGURATION_RESOURCES
static String	ORC_MIME_TYPE
private org.apache.hadoop.conf.Configuration	orcConfig
private static List<PropertyDescriptor>	propertyDescriptors
static String	RECORD_COUNT_ATTRIBUTE
(package private) static Relationship	REL_FAILURE
(package private) static Relationship	REL_SUCCESS
private static Set<Relationship>	relationships
static PropertyDescriptor	STRIPE_SIZE

Constructor Summary

Constructors

Constructor and Description
ConvertAvroToORC()

Method Summary

Modifier and Type	Method and Description
Set<Relationship>	getRelationships()
protected List<PropertyDescriptor>	getSupportedPropertyDescriptors()
void	onTrigger(ProcessContext context, ProcessSession session)
void	setup(ProcessContext context)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

customValidate, equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.processor.Processor

isStateful

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Field Detail

ORC_MIME_TYPE

public static final String ORC_MIME_TYPE

See Also:

Constant Field Values

HIVE_DDL_ATTRIBUTE

public static final String HIVE_DDL_ATTRIBUTE

See Also:

Constant Field Values

RECORD_COUNT_ATTRIBUTE

public static final String RECORD_COUNT_ATTRIBUTE

See Also:

Constant Field Values

ORC_CONFIGURATION_RESOURCES

public static final PropertyDescriptor ORC_CONFIGURATION_RESOURCES

STRIPE_SIZE

public static final PropertyDescriptor STRIPE_SIZE

BUFFER_SIZE

public static final PropertyDescriptor BUFFER_SIZE

COMPRESSION_TYPE

public static final PropertyDescriptor COMPRESSION_TYPE

HIVE_TABLE_NAME

public static final PropertyDescriptor HIVE_TABLE_NAME

REL_SUCCESS

static final Relationship REL_SUCCESS

REL_FAILURE

static final Relationship REL_FAILURE

propertyDescriptors

private static final List<PropertyDescriptor> propertyDescriptors

relationships

private static final Set<Relationship> relationships

orcConfig

private volatile org.apache.hadoop.conf.Configuration orcConfig

Constructor Detail

ConvertAvroToORC

public ConvertAvroToORC()

Method Detail

getSupportedPropertyDescriptors

protected List<PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractConfigurableComponent

getRelationships

public Set<Relationship> getRelationships()

Specified by:

getRelationships in interface Processor

Overrides:

getRelationships in class AbstractSessionFactoryProcessor

setup

@OnScheduled
public void setup(ProcessContext context)

onTrigger

public void onTrigger(ProcessContext context,
                      ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException