Class CaptureChangeMySQL
java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL
- All Implemented Interfaces:
ConfigurableComponent,Processor
@TriggerSerially
@PrimaryNodeOnly
@InputRequirement(INPUT_FORBIDDEN)
@Tags({"sql","jdbc","cdc","mysql","transaction","event"})
@CapabilityDescription("Retrieves Change Data Capture (CDC) events from a MySQL database. CDC Events include INSERT, UPDATE, DELETE operations. Events are output as either a group of a specified number of events (the default is 1 so each event becomes its own flow file) or grouped as a full transaction (BEGIN to COMMIT). All events are ordered by the time at which the operation occurred. NOTE: If the processor is stopped before the specified number of events have been written to a flow file, the partial flow file will be output in order to maintain the consistency of the event stream.")
@Stateful(scopes=CLUSTER,
description="Information such as a \'pointer\' to the current CDC event in the database is stored by this processor, such that it can continue from the same location if restarted.")
@WritesAttribute(attribute="cdc.sequence.id",description="A sequence identifier (i.e. strictly increasing integer value) specifying the order of the CDC event flow file relative to the other event flow file(s).") @WritesAttribute(attribute="cdc.event.type",description="A string indicating the type of CDC event that occurred, including (but not limited to) \'begin\', \'insert\', \'update\', \'delete\', \'ddl\' and \'commit\'.") @WritesAttribute(attribute="mime.type",description="The processor outputs flow file content in JSON format, and sets the mime.type attribute to application/json")
@RequiresInstanceClassLoading
@Restricted(restrictions=@Restriction(requiredPermission=REFERENCE_REMOTE_RESOURCES,explanation="Database Driver Location can reference resources over HTTP"))
public class CaptureChangeMySQL
extends AbstractSessionFactoryProcessor
A processor to retrieve Change Data Capture (CDC) events and send them as flow files.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic classstatic classprivate static classprivate class -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final BeginEventHandlerprivate com.github.shyiko.mysql.binlog.BinaryLogClientprivate final CaptureChangeMySQL.BinlogEventStateprivate final CommitEventHandlerstatic final PropertyDescriptorprivate DataCaptureStateprivate intprivate ProcessSessionstatic final PropertyDescriptorprivate Patternprivate final DDLEventHandlerprivate static final intprivate final DeleteEventHandlerstatic final PropertyDescriptorprivate static final intstatic final PropertyDescriptorstatic final PropertyDescriptorprivate BinlogEventListenerstatic final PropertyDescriptorprivate EventWriterConfigurationprivate com.github.shyiko.mysql.binlog.GtidSetstatic final PropertyDescriptorstatic final PropertyDescriptorstatic final PropertyDescriptorstatic final PropertyDescriptorstatic final PropertyDescriptorstatic final PropertyDescriptorstatic final PropertyDescriptorprivate final InsertEventHandlerprivate BinlogLifecycleListenerprivate static final Patternstatic final PropertyDescriptorstatic final PropertyDescriptorprivate static final List<PropertyDescriptor> private final BlockingQueue<RawBinlogEvent> static final Relationshipprotected static Set<Relationship> static final PropertyDescriptorstatic final PropertyDescriptorprivate booleanstatic final PropertyDescriptorstatic final PropertyDescriptorprivate static final AllowableValueprivate static final AllowableValueprivate static final AllowableValueprivate static final AllowableValuestatic final PropertyDescriptorstatic final PropertyDescriptorprivate final Map<TableInfoCacheKey, TableInfo> private Patternprivate final UpdateEventHandlerstatic final PropertyDescriptorstatic final PropertyDescriptor -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected voidprotected voidconnect(List<InetSocketAddress> hosts, String username, String password, Long serverId, String driverLocation, String driverName, long connectTimeout, SSLContextService sslContextService, com.github.shyiko.mysql.binlog.network.SSLMode sslMode) protected com.github.shyiko.mysql.binlog.BinaryLogClientcreateBinlogClient(String hostname, int port, String username, String password) (package private) BinlogEventListenercreateBinlogEventListener(com.github.shyiko.mysql.binlog.BinaryLogClient client, BlockingQueue<RawBinlogEvent> q) Creates and returns a BinlogEventListener instance, associated with the specified binlog client and event queue.(package private) BinlogLifecycleListenerCreates and returns a BinlogLifecycleListener instance, associated with the specified binlog client and event queue.protected Collection<ValidationResult> customValidate(ValidationContext validationContext) private List<InetSocketAddress> Get a list of hosts from a NiFi property, e.g.protected Connectionprotected List<PropertyDescriptor> private booleanisQueryDDL(String sql) protected TableInfoRetrieves the column information for the specified database and table.protected StringnormalizeQuery(String sql) voidvoidonTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) voidoutputEvents(ProcessSession session, ProcessContext context, ComponentLog log) protected voidregisterDriver(String locationString, String drvName) using Thread.currentThread().getContextClassLoader(); will ensure that you are using the ClassLoader for you NAR.voidsetup(ProcessContext context) voidstop()private voidupdateState(ProcessSession session, DataCaptureState dataCaptureState) private voidupdateState(ProcessSession session, DataCaptureState dataCaptureState, boolean inTransaction) Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor
getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrueMethods inherited from class org.apache.nifi.components.AbstractConfigurableComponent
equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validateMethods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, waitMethods inherited from interface org.apache.nifi.components.ConfigurableComponent
getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validateMethods inherited from interface org.apache.nifi.processor.Processor
isStateful, migrateProperties, migrateRelationships
-
Field Details
-
DO_NOT_SET
private static final int DO_NOT_SET- See Also:
-
DEFAULT_MYSQL_PORT
private static final int DEFAULT_MYSQL_PORT- See Also:
-
MULTI_COMMENT_PATTERN
-
REL_SUCCESS
-
relationships
-
SSL_MODE_DISABLED
-
SSL_MODE_PREFERRED
-
SSL_MODE_REQUIRED
-
SSL_MODE_VERIFY_IDENTITY
-
DATABASE_NAME_PATTERN
-
TABLE_NAME_PATTERN
-
CONNECT_TIMEOUT
-
HOSTS
-
DRIVER_NAME
-
DRIVER_LOCATION
-
USERNAME
-
PASSWORD
-
EVENTS_PER_FLOWFILE_STRATEGY
-
NUMBER_OF_EVENTS_PER_FLOWFILE
-
SERVER_ID
-
DIST_CACHE_CLIENT
-
RETRIEVE_ALL_RECORDS
-
INCLUDE_BEGIN_COMMIT
-
INCLUDE_DDL_EVENTS
-
STATE_UPDATE_INTERVAL
-
INIT_SEQUENCE_ID
-
INIT_BINLOG_FILENAME
-
INIT_BINLOG_POSITION
-
USE_BINLOG_GTID
-
INIT_BINLOG_GTID
-
SSL_MODE
-
SSL_CONTEXT_SERVICE
-
propDescriptors
-
binlogClient
private volatile com.github.shyiko.mysql.binlog.BinaryLogClient binlogClient -
eventListener
-
lifecycleListener
-
gtidSet
private volatile com.github.shyiko.mysql.binlog.GtidSet gtidSet -
queue
-
tableInfoCache
-
currentSession
-
currentDataCaptureState
-
binlogResourceInfo
-
databaseNamePattern
-
tableNamePattern
-
skipTable
private volatile boolean skipTable -
currentHost
private int currentHost -
jdbcConnectionHolder
-
binlogEventState
-
beginEventHandler
-
commitEventHandler
-
ddlEventHandler
-
insertEventHandler
-
deleteEventHandler
-
updateEventHandler
-
eventWriterConfiguration
-
-
Constructor Details
-
CaptureChangeMySQL
public CaptureChangeMySQL()
-
-
Method Details
-
getRelationships
- Specified by:
getRelationshipsin interfaceProcessor- Overrides:
getRelationshipsin classAbstractSessionFactoryProcessor
-
getSupportedPropertyDescriptors
- Overrides:
getSupportedPropertyDescriptorsin classAbstractConfigurableComponent
-
customValidate
- Overrides:
customValidatein classAbstractConfigurableComponent
-
onPrimaryNodeChange
@OnPrimaryNodeStateChange public void onPrimaryNodeChange(PrimaryNodeState state) throws CDCException - Throws:
CDCException
-
setup
-
onTrigger
public void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException - Throws:
ProcessException
-
getHosts
Get a list of hosts from a NiFi property, e.g.- Parameters:
hostsString- A comma-separated list of hosts (host:port,host2:port2, etc.)- Returns:
- List of InetSocketAddresses for the hosts
-
connect
protected void connect(List<InetSocketAddress> hosts, String username, String password, Long serverId, String driverLocation, String driverName, long connectTimeout, SSLContextService sslContextService, com.github.shyiko.mysql.binlog.network.SSLMode sslMode) throws IOException - Throws:
IOException
-
outputEvents
public void outputEvents(ProcessSession session, ProcessContext context, ComponentLog log) throws IOException - Throws:
IOException
-
isQueryDDL
-
clearState
- Throws:
IOException
-
normalizeQuery
-
stop
- Throws:
CDCException
-
updateState
private void updateState(ProcessSession session, DataCaptureState dataCaptureState) throws IOException - Throws:
IOException
-
updateState
private void updateState(ProcessSession session, DataCaptureState dataCaptureState, boolean inTransaction) throws IOException - Throws:
IOException
-
createBinlogEventListener
BinlogEventListener createBinlogEventListener(com.github.shyiko.mysql.binlog.BinaryLogClient client, BlockingQueue<RawBinlogEvent> q) Creates and returns a BinlogEventListener instance, associated with the specified binlog client and event queue.- Parameters:
client- A reference to a BinaryLogClient. The listener is associated with the given client, such that the listener is notified when events are available to the given client.q- A queue used to communicate events between the listener and the NiFi processor thread.- Returns:
- A BinlogEventListener instance, which will be notified of events associated with the specified client
-
createBinlogLifecycleListener
BinlogLifecycleListener createBinlogLifecycleListener()Creates and returns a BinlogLifecycleListener instance, associated with the specified binlog client and event queue.- Returns:
- A BinlogLifecycleListener instance, which will be notified of events associated with the specified client
-
createBinlogClient
-
loadTableInfo
Retrieves the column information for the specified database and table. The column information can be used to enrich CDC events coming from the RDBMS.- Parameters:
key- A TableInfoCacheKey reference, which contains the database and table names- Returns:
- A TableInfo instance with the ColumnDefinitions provided (if retrieved successfully from the database)
- Throws:
SQLException
-
getJdbcConnection
- Throws:
SQLException
-
registerDriver
using Thread.currentThread().getContextClassLoader(); will ensure that you are using the ClassLoader for you NAR.- Throws:
InitializationException- if there is a problem obtaining the ClassLoader
-