Package org.apache.nifi.cdc.mysql.processors

Class CaptureChangeMySQL

java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL
All Implemented Interfaces:
org.apache.nifi.components.ConfigurableComponent, org.apache.nifi.processor.Processor
@TriggerSerially @PrimaryNodeOnly @InputRequirement(INPUT_FORBIDDEN) @Tags({"sql","jdbc","cdc","mysql","transaction","event"}) @CapabilityDescription("Retrieves Change Data Capture (CDC) events from a MySQL database. CDC Events include INSERT, UPDATE, DELETE operations. Events are output as either a group of a specified number of events (the default is 1 so each event becomes its own flow file) or grouped as a full transaction (BEGIN to COMMIT). All events are ordered by the time at which the operation occurred. NOTE: If the processor is stopped before the specified number of events have been written to a flow file, the partial flow file will be output in order to maintain the consistency of the event stream.") @Stateful(scopes=CLUSTER, description="Information such as a \'pointer\' to the current CDC event in the database is stored by this processor, such that it can continue from the same location if restarted.") @WritesAttribute(attribute="cdc.sequence.id",description="A sequence identifier (i.e. strictly increasing integer value) specifying the order of the CDC event flow file relative to the other event flow file(s).") @WritesAttribute(attribute="cdc.event.type",description="A string indicating the type of CDC event that occurred, including (but not limited to) \'begin\', \'insert\', \'update\', \'delete\', \'ddl\' and \'commit\'.") @WritesAttribute(attribute="mime.type",description="The processor outputs flow file content in JSON format, and sets the mime.type attribute to application/json") @RequiresInstanceClassLoading @Restricted(restrictions=@Restriction(requiredPermission=REFERENCE_REMOTE_RESOURCES,explanation="Database Driver Location can reference resources over HTTP")) public class CaptureChangeMySQL extends org.apache.nifi.processor.AbstractSessionFactoryProcessor
A processor to retrieve Change Data Capture (CDC) events and send them as flow files.

  • Field Details

    • DO_NOT_SET

      private static final int DO_NOT_SET
      See Also:

    • DEFAULT_MYSQL_PORT

      private static final int DEFAULT_MYSQL_PORT
      See Also:

    • MULTI_COMMENT_PATTERN

      private static final Pattern MULTI_COMMENT_PATTERN

    • REL_SUCCESS

      public static final org.apache.nifi.processor.Relationship REL_SUCCESS

    • relationships

      protected static Set<org.apache.nifi.processor.Relationship> relationships

    • SSL_MODE_DISABLED

      private static final org.apache.nifi.components.AllowableValue SSL_MODE_DISABLED

    • SSL_MODE_PREFERRED

      private static final org.apache.nifi.components.AllowableValue SSL_MODE_PREFERRED

    • SSL_MODE_REQUIRED

      private static final org.apache.nifi.components.AllowableValue SSL_MODE_REQUIRED

    • SSL_MODE_VERIFY_IDENTITY

      private static final org.apache.nifi.components.AllowableValue SSL_MODE_VERIFY_IDENTITY

    • DATABASE_NAME_PATTERN

      public static final org.apache.nifi.components.PropertyDescriptor DATABASE_NAME_PATTERN

    • TABLE_NAME_PATTERN

      public static final org.apache.nifi.components.PropertyDescriptor TABLE_NAME_PATTERN

    • CONNECT_TIMEOUT

      public static final org.apache.nifi.components.PropertyDescriptor CONNECT_TIMEOUT

    • HOSTS

      public static final org.apache.nifi.components.PropertyDescriptor HOSTS

    • DRIVER_NAME

      public static final org.apache.nifi.components.PropertyDescriptor DRIVER_NAME

    • DRIVER_LOCATION

      public static final org.apache.nifi.components.PropertyDescriptor DRIVER_LOCATION

    • USERNAME

      public static final org.apache.nifi.components.PropertyDescriptor USERNAME

    • PASSWORD

      public static final org.apache.nifi.components.PropertyDescriptor PASSWORD

    • EVENTS_PER_FLOWFILE_STRATEGY

      public static final org.apache.nifi.components.PropertyDescriptor EVENTS_PER_FLOWFILE_STRATEGY

    • NUMBER_OF_EVENTS_PER_FLOWFILE

      public static final org.apache.nifi.components.PropertyDescriptor NUMBER_OF_EVENTS_PER_FLOWFILE

    • SERVER_ID

      public static final org.apache.nifi.components.PropertyDescriptor SERVER_ID

    • DIST_CACHE_CLIENT

      public static final org.apache.nifi.components.PropertyDescriptor DIST_CACHE_CLIENT

    • RETRIEVE_ALL_RECORDS

      public static final org.apache.nifi.components.PropertyDescriptor RETRIEVE_ALL_RECORDS

    • INCLUDE_BEGIN_COMMIT

      public static final org.apache.nifi.components.PropertyDescriptor INCLUDE_BEGIN_COMMIT

    • INCLUDE_DDL_EVENTS

      public static final org.apache.nifi.components.PropertyDescriptor INCLUDE_DDL_EVENTS

    • INIT_SEQUENCE_ID

      public static final org.apache.nifi.components.PropertyDescriptor INIT_SEQUENCE_ID

    • INIT_BINLOG_FILENAME

      public static final org.apache.nifi.components.PropertyDescriptor INIT_BINLOG_FILENAME

    • INIT_BINLOG_POSITION

      public static final org.apache.nifi.components.PropertyDescriptor INIT_BINLOG_POSITION

    • USE_BINLOG_GTID

      public static final org.apache.nifi.components.PropertyDescriptor USE_BINLOG_GTID

    • INIT_BINLOG_GTID

      public static final org.apache.nifi.components.PropertyDescriptor INIT_BINLOG_GTID

    • SSL_MODE

      public static final org.apache.nifi.components.PropertyDescriptor SSL_MODE

    • SSL_CONTEXT_SERVICE

      public static final org.apache.nifi.components.PropertyDescriptor SSL_CONTEXT_SERVICE

    • propDescriptors

      private static final List<org.apache.nifi.components.PropertyDescriptor> propDescriptors

    • binlogClient

      private volatile com.github.shyiko.mysql.binlog.BinaryLogClient binlogClient

    • eventListener

      private volatile BinlogEventListener eventListener

    • lifecycleListener

      private volatile BinlogLifecycleListener lifecycleListener

    • gtidSet

      private volatile com.github.shyiko.mysql.binlog.GtidSet gtidSet

    • queue

      private final BlockingQueue<RawBinlogEvent> queue

    • tableInfoCache

      private final Map<TableInfoCacheKey,TableInfo> tableInfoCache

    • currentSession

      private volatile org.apache.nifi.processor.ProcessSession currentSession

    • currentDataCaptureState

      private DataCaptureState currentDataCaptureState

    • binlogResourceInfo

      private volatile CaptureChangeMySQL.BinlogResourceInfo binlogResourceInfo

    • databaseNamePattern

      private volatile Pattern databaseNamePattern

    • tableNamePattern

      private volatile Pattern tableNamePattern

    • skipTable

      private volatile boolean skipTable

    • currentHost

      private int currentHost

    • jdbcConnectionHolder

      private volatile CaptureChangeMySQL.JDBCConnectionHolder jdbcConnectionHolder

    • binlogEventState

      private final CaptureChangeMySQL.BinlogEventState binlogEventState

    • beginEventHandler

      private final BeginEventHandler beginEventHandler

    • commitEventHandler

      private final CommitEventHandler commitEventHandler

    • ddlEventHandler

      private final DDLEventHandler ddlEventHandler

    • insertEventHandler

      private final InsertEventHandler insertEventHandler

    • deleteEventHandler

      private final DeleteEventHandler deleteEventHandler

    • updateEventHandler

      private final UpdateEventHandler updateEventHandler

    • eventWriterConfiguration

      private volatile EventWriterConfiguration eventWriterConfiguration

  • Constructor Details

    • CaptureChangeMySQL

      public CaptureChangeMySQL()

  • Method Details

    • getRelationships

      public Set<org.apache.nifi.processor.Relationship> getRelationships()
      Specified by:
      getRelationships in interface org.apache.nifi.processor.Processor
      Overrides:
      getRelationships in class org.apache.nifi.processor.AbstractSessionFactoryProcessor

    • getSupportedPropertyDescriptors

      protected List<org.apache.nifi.components.PropertyDescriptor> getSupportedPropertyDescriptors()
      Overrides:
      getSupportedPropertyDescriptors in class org.apache.nifi.components.AbstractConfigurableComponent

    • migrateProperties

      public void migrateProperties(org.apache.nifi.migration.PropertyConfiguration config)

    • customValidate

      protected Collection<org.apache.nifi.components.ValidationResult> customValidate(org.apache.nifi.components.ValidationContext validationContext)
      Overrides:
      customValidate in class org.apache.nifi.components.AbstractConfigurableComponent

    • onPrimaryNodeChange

      @OnPrimaryNodeStateChange public void onPrimaryNodeChange(org.apache.nifi.annotation.notification.PrimaryNodeState state) throws CDCException
      Throws:
      CDCException

    • setup

      public void setup(org.apache.nifi.processor.ProcessContext context)

    • onTrigger

      public void onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSessionFactory sessionFactory) throws org.apache.nifi.processor.exception.ProcessException
      Throws:
      org.apache.nifi.processor.exception.ProcessException

    • getHosts

      private List<InetSocketAddress> getHosts(String hostsString)
      Get a list of hosts from a NiFi property, e.g.
      Parameters:
      hostsString - A comma-separated list of hosts (host:port,host2:port2, etc.)
      Returns:
      List of InetSocketAddresses for the hosts

    • connect

      protected void connect(List<InetSocketAddress> hosts, String username, String password, Long serverId, String driverLocation, String driverName, long connectTimeout, SSLContextService sslContextService, com.github.shyiko.mysql.binlog.network.SSLMode sslMode) throws IOException
      Throws:
      IOException

    • outputEvents

      public void outputEvents(org.apache.nifi.processor.ProcessSession session, org.apache.nifi.processor.ProcessContext context, org.apache.nifi.logging.ComponentLog log) throws IOException
      Throws:
      IOException

    • isQueryDDL

      private boolean isQueryDDL(String sql)

    • clearState

      protected void clearState() throws IOException
      Throws:
      IOException

    • normalizeQuery

      protected String normalizeQuery(String sql)

    • stop

      @OnStopped public void stop() throws CDCException
      Throws:
      CDCException

    • updateState

      private void updateState(org.apache.nifi.processor.ProcessSession session, DataCaptureState dataCaptureState) throws IOException
      Throws:
      IOException

    • updateState

      private void updateState(org.apache.nifi.processor.ProcessSession session, DataCaptureState dataCaptureState, boolean inTransaction) throws IOException
      Throws:
      IOException

    • createBinlogEventListener

      BinlogEventListener createBinlogEventListener(com.github.shyiko.mysql.binlog.BinaryLogClient client, BlockingQueue<RawBinlogEvent> q)
      Creates and returns a BinlogEventListener instance, associated with the specified binlog client and event queue.
      Parameters:
      client - A reference to a BinaryLogClient. The listener is associated with the given client, such that the listener is notified when events are available to the given client.
      q - A queue used to communicate events between the listener and the NiFi processor thread.
      Returns:
      A BinlogEventListener instance, which will be notified of events associated with the specified client

    • createBinlogLifecycleListener

      BinlogLifecycleListener createBinlogLifecycleListener()
      Creates and returns a BinlogLifecycleListener instance, associated with the specified binlog client and event queue.
      Returns:
      A BinlogLifecycleListener instance, which will be notified of events associated with the specified client

    • createBinlogClient

      protected com.github.shyiko.mysql.binlog.BinaryLogClient createBinlogClient(String hostname, int port, String username, String password)

    • loadTableInfo

      protected TableInfo loadTableInfo(TableInfoCacheKey key) throws SQLException
      Retrieves the column information for the specified database and table. The column information can be used to enrich CDC events coming from the RDBMS.
      Parameters:
      key - A TableInfoCacheKey reference, which contains the database and table names
      Returns:
      A TableInfo instance with the ColumnDefinitions provided (if retrieved successfully from the database)
      Throws:
      SQLException

    • getJdbcConnection

      protected Connection getJdbcConnection() throws SQLException
      Throws:
      SQLException

    • registerDriver

      protected void registerDriver(String locationString, String drvName) throws InitializationException
      using Thread.currentThread().getContextClassLoader(); will ensure that you are using the ClassLoader for you NAR.
      Throws:
      InitializationException - if there is a problem obtaining the ClassLoader