org.apache.nifi.processors.kudu

Class AbstractKuduProcessor

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.kudu.AbstractKuduProcessor

All Implemented Interfaces:

ConfigurableComponent, Processor

Direct Known Subclasses:

PutKudu

public abstract class AbstractKuduProcessor
extends AbstractProcessor

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	AbstractKuduProcessor.ClientThreadFactory

Field Summary

Fields

Modifier and Type	Field and Description
private static int	DEFAULT_WORKER_COUNT
(package private) static PropertyDescriptor	KERBEROS_CREDENTIALS_SERVICE
(package private) static PropertyDescriptor	KERBEROS_PASSWORD
(package private) static PropertyDescriptor	KERBEROS_PRINCIPAL
(package private) static PropertyDescriptor	KERBEROS_USER_SERVICE
private KerberosUser	kerberosUser
(package private) static PropertyDescriptor	KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS
(package private) static PropertyDescriptor	KUDU_MASTERS
(package private) static PropertyDescriptor	KUDU_OPERATION_TIMEOUT_MS
(package private) static PropertyDescriptor	KUDU_SASL_PROTOCOL_NAME
private org.apache.kudu.client.KuduClient	kuduClient
private Lock	kuduClientReadLock
private ReadWriteLock	kuduClientReadWriteLock
private Lock	kuduClientWriteLock
private static String	MICROSECOND_TIMESTAMP_PATTERN Timestamp Pattern overrides default RecordFieldType.TIMESTAMP pattern of yyyy-MM-dd HH:mm:ss with optional microseconds
private static FieldConverter<Object,Timestamp>	TIMESTAMP_FIELD_CONVERTER
(package private) static PropertyDescriptor	WORKER_COUNT

Constructor Summary

Constructors

Constructor and Description
AbstractKuduProcessor()

Method Summary

Modifier and Type	Method and Description
protected org.apache.kudu.client.KuduClient	buildClient(ProcessContext context)
protected void	buildPartialRow(org.apache.kudu.Schema schema, org.apache.kudu.client.PartialRow row, Record record, List<String> fieldNames, boolean ignoreNull, boolean lowercaseFields)
protected KerberosUser	createKerberosKeytabUser(String principal, String keytab, ProcessContext context)
protected KerberosUser	createKerberosPasswordUser(String principal, String password, ProcessContext context)
protected void	createKerberosUserAndOrKuduClient(ProcessContext context)
protected void	createKuduClient(ProcessContext context)
protected Collection<ValidationResult>	customValidate(ValidationContext context)
protected void	executeOnKuduClient(Consumer<org.apache.kudu.client.KuduClient> actionOnKuduClient)
protected void	flushKuduSession(org.apache.kudu.client.KuduSession kuduSession, boolean close, List<org.apache.kudu.client.RowError> rowErrors)
protected org.apache.kudu.client.AlterTableOptions	getAddNullableColumnStatement(String columnName, DataType nifiType) Based on NiFi field declaration, generates an alter statement to extend table with new column.
private Date	getDate(Object value, String recordFieldName, String format) Get java.sql.Date from Record Field Value with optional parsing when input value is a String
protected KerberosUser	getKerberosUser()
private org.apache.kudu.ColumnTypeAttributes	getKuduTypeAttributes(DataType nifiType)
private Optional<String>	getTimestampPattern(Optional<DataType> optionalDataType) Get Timestamp Pattern and override Timestamp Record Field pattern with optional microsecond pattern
void	shutdown()
protected boolean	supportsIgnoreOperations()
private org.apache.kudu.Type	toKuduType(DataType nifiType) Converts a NiFi DataType to it's equivalent Kudu Type.

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger, onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, getRelationships, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, getSupportedPropertyDescriptors, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.processor.Processor

isStateful

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Field Detail

KUDU_MASTERS

static final PropertyDescriptor KUDU_MASTERS

KERBEROS_CREDENTIALS_SERVICE

static final PropertyDescriptor KERBEROS_CREDENTIALS_SERVICE

KERBEROS_USER_SERVICE

static final PropertyDescriptor KERBEROS_USER_SERVICE

KERBEROS_PRINCIPAL

static final PropertyDescriptor KERBEROS_PRINCIPAL

KERBEROS_PASSWORD

static final PropertyDescriptor KERBEROS_PASSWORD

KUDU_OPERATION_TIMEOUT_MS

static final PropertyDescriptor KUDU_OPERATION_TIMEOUT_MS

KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS

static final PropertyDescriptor KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS

DEFAULT_WORKER_COUNT

private static final int DEFAULT_WORKER_COUNT

WORKER_COUNT

static final PropertyDescriptor WORKER_COUNT

KUDU_SASL_PROTOCOL_NAME

static final PropertyDescriptor KUDU_SASL_PROTOCOL_NAME

TIMESTAMP_FIELD_CONVERTER

private static final FieldConverter<Object,Timestamp> TIMESTAMP_FIELD_CONVERTER

MICROSECOND_TIMESTAMP_PATTERN

private static final String MICROSECOND_TIMESTAMP_PATTERN

Timestamp Pattern overrides default RecordFieldType.TIMESTAMP pattern of yyyy-MM-dd HH:mm:ss with optional microseconds

See Also:

Constant Field Values

kuduClient

private volatile org.apache.kudu.client.KuduClient kuduClient

kuduClientReadWriteLock

private final ReadWriteLock kuduClientReadWriteLock

kuduClientReadLock

private final Lock kuduClientReadLock

kuduClientWriteLock

private final Lock kuduClientWriteLock

kerberosUser

private volatile KerberosUser kerberosUser

Constructor Detail

AbstractKuduProcessor

public AbstractKuduProcessor()

Method Detail

getKerberosUser

protected KerberosUser getKerberosUser()

supportsIgnoreOperations

protected boolean supportsIgnoreOperations()

createKerberosUserAndOrKuduClient

protected void createKerberosUserAndOrKuduClient(ProcessContext context)

createKuduClient

protected void createKuduClient(ProcessContext context)

buildClient

protected org.apache.kudu.client.KuduClient buildClient(ProcessContext context)

executeOnKuduClient

protected void executeOnKuduClient(Consumer<org.apache.kudu.client.KuduClient> actionOnKuduClient)

flushKuduSession

protected void flushKuduSession(org.apache.kudu.client.KuduSession kuduSession,
                                boolean close,
                                List<org.apache.kudu.client.RowError> rowErrors)
                         throws org.apache.kudu.client.KuduException

Throws:

org.apache.kudu.client.KuduException

createKerberosKeytabUser

protected KerberosUser createKerberosKeytabUser(String principal,
                                                String keytab,
                                                ProcessContext context)

createKerberosPasswordUser

protected KerberosUser createKerberosPasswordUser(String principal,
                                                  String password,
                                                  ProcessContext context)

customValidate

protected Collection<ValidationResult> customValidate(ValidationContext context)

Overrides:

customValidate in class AbstractConfigurableComponent

shutdown

@OnStopped
public void shutdown()
                         throws Exception

Throws:

Exception

buildPartialRow

protected void buildPartialRow(org.apache.kudu.Schema schema,
                               org.apache.kudu.client.PartialRow row,
                               Record record,
                               List<String> fieldNames,
                               boolean ignoreNull,
                               boolean lowercaseFields)

getTimestampPattern

private Optional<String> getTimestampPattern(Optional<DataType> optionalDataType)

Get Timestamp Pattern and override Timestamp Record Field pattern with optional microsecond pattern

Parameters:

optionalDataType - Optional Data Type

Returns:

Optional Timestamp Pattern

getDate

private Date getDate(Object value,
                     String recordFieldName,
                     String format)

Get java.sql.Date from Record Field Value with optional parsing when input value is a String

Parameters:

value - Record Field Value

recordFieldName - Record Field Name

format - Date Format Pattern

Returns:

Date object or null when value is null

toKuduType

private org.apache.kudu.Type toKuduType(DataType nifiType)

Converts a NiFi DataType to it's equivalent Kudu Type.

getKuduTypeAttributes

private org.apache.kudu.ColumnTypeAttributes getKuduTypeAttributes(DataType nifiType)

getAddNullableColumnStatement

protected org.apache.kudu.client.AlterTableOptions getAddNullableColumnStatement(String columnName,
                                                                                 DataType nifiType)

Based on NiFi field declaration, generates an alter statement to extend table with new column. Note: simply calling AlterTableOptions.addNullableColumn(String, Type) is not sufficient as it does not cover BigDecimal scale and precision handling.

Parameters:

columnName - Name of the new table column.

nifiType - Type of the field.

Returns:

Alter table statement to extend table with the new field.