Package org.apache.nifi.processors.pulsar.pubsub

Class ConsumePulsarRecord

java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.processor.AbstractProcessor
org.apache.nifi.processors.pulsar.AbstractPulsarConsumerProcessor<org.apache.pulsar.client.api.schema.GenericRecord>
org.apache.nifi.processors.pulsar.pubsub.ConsumePulsarRecord
All Implemented Interfaces:
org.apache.nifi.components.ConfigurableComponent, org.apache.nifi.processor.Processor
@CapabilityDescription("Consumes messages from Apache Pulsar. The complementary NiFi processor for sending messages is PublishPulsarRecord. Please note that, at this time, the Processor assumes that all records that are retrieved have the same schema. If any of the Pulsar messages that are pulled but cannot be parsed or written with the configured Record Reader or Record Writer, the contents of the message will be written to a separate FlowFile, and that FlowFile will be transferred to the \'parse.failure\' relationship. Otherwise, each FlowFile is sent to the \'success\' relationship and may contain many individual messages within the single FlowFile. A \'record.count\' attribute is added to indicate how many messages are contained in the FlowFile. No two Pulsar messages will be placed into the same FlowFile if they have different schemas.") @Tags({"Pulsar","Get","Record","csv","avro","json","Ingest","Ingress","Topic","PubSub","Consume"}) @WritesAttributes(@WritesAttribute(attribute="record.count",description="The number of records received")) @InputRequirement(INPUT_FORBIDDEN) @SeeAlso({PublishPulsar.class,ConsumePulsar.class,PublishPulsarRecord.class}) public class ConsumePulsarRecord extends AbstractPulsarConsumerProcessor<org.apache.pulsar.client.api.schema.GenericRecord>

  • Field Details

    • MSG_COUNT

      public static final String MSG_COUNT
      See Also:

    • RECORD_SEPARATOR

      private static final String RECORD_SEPARATOR
      See Also:

    • RECORD_READER

      public static final org.apache.nifi.components.PropertyDescriptor RECORD_READER

    • RECORD_WRITER

      public static final org.apache.nifi.components.PropertyDescriptor RECORD_WRITER

    • MAX_WAIT_TIME

      public static final org.apache.nifi.components.PropertyDescriptor MAX_WAIT_TIME

    • REL_PARSE_FAILURE

      public static final org.apache.nifi.processor.Relationship REL_PARSE_FAILURE

    • PROPERTIES

      private static final List<org.apache.nifi.components.PropertyDescriptor> PROPERTIES

    • RELATIONSHIPS

      private static final Set<org.apache.nifi.processor.Relationship> RELATIONSHIPS

  • Constructor Details

    • ConsumePulsarRecord

      public ConsumePulsarRecord()

  • Method Details

    • getRelationships

      public Set<org.apache.nifi.processor.Relationship> getRelationships()
      Specified by:
      getRelationships in interface org.apache.nifi.processor.Processor
      Overrides:
      getRelationships in class AbstractPulsarConsumerProcessor<org.apache.pulsar.client.api.schema.GenericRecord>

    • getSupportedPropertyDescriptors

      protected List<org.apache.nifi.components.PropertyDescriptor> getSupportedPropertyDescriptors()
      Overrides:
      getSupportedPropertyDescriptors in class AbstractPulsarConsumerProcessor<org.apache.pulsar.client.api.schema.GenericRecord>

    • onTrigger

      public void onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session) throws org.apache.nifi.processor.exception.ProcessException
      Specified by:
      onTrigger in class org.apache.nifi.processor.AbstractProcessor
      Throws:
      org.apache.nifi.processor.exception.ProcessException

    • getMessages

      private List<org.apache.pulsar.client.api.Message<org.apache.pulsar.client.api.schema.GenericRecord>> getMessages(org.apache.pulsar.client.api.Consumer<org.apache.pulsar.client.api.schema.GenericRecord> consumer, int maxMessages) throws org.apache.pulsar.client.api.PulsarClientException
      Retrieve a batch of up to maxMessages for processing.
      Parameters:
      consumer - - The Pulsar consumer.
      maxMessages - - The maximum number of messages to consume from Pulsar.
      Returns:
      A List of Messages
      Throws:
      org.apache.pulsar.client.api.PulsarClientException - in the event we cannot communicate with the Pulsar broker.

    • consumeMessages

      private void consumeMessages(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session, org.apache.pulsar.client.api.Consumer<org.apache.pulsar.client.api.schema.GenericRecord> consumer, List<org.apache.pulsar.client.api.Message<org.apache.pulsar.client.api.schema.GenericRecord>> messages, org.apache.nifi.serialization.RecordReaderFactory readerFactory, org.apache.nifi.serialization.RecordSetWriterFactory writerFactory, byte[] demarcator, boolean async) throws org.apache.pulsar.client.api.PulsarClientException
      Perform the actual processing of the messages, by parsing the messages and writing them out to a FlowFile. All of the messages passed in shall be routed to either SUCCESS or PARSE_FAILURE, allowing us to acknowledge the receipt of the messages to Pulsar, so they are not re-sent.
      Parameters:
      context - - The current ProcessContext
      session - - The current ProcessSession.
      consumer - - The Pulsar consumer.
      messages - - A list of messages.
      readerFactory - - The factory used to read the messages.
      writerFactory - - The factory used to write the messages.
      demarcator - - The value used to identify unique records in the list
      async - - Whether or not to consume the messages asynchronously.
      Throws:
      org.apache.pulsar.client.api.PulsarClientException - if there is an issue communicating with Apache Pulsar.

    • acknowledge

      private void acknowledge(org.apache.pulsar.client.api.Consumer<org.apache.pulsar.client.api.schema.GenericRecord> consumer, org.apache.pulsar.client.api.Message<org.apache.pulsar.client.api.schema.GenericRecord> msg, boolean async) throws org.apache.pulsar.client.api.PulsarClientException
      Throws:
      org.apache.pulsar.client.api.PulsarClientException

    • acknowledgeCumulative

      private void acknowledgeCumulative(org.apache.pulsar.client.api.Consumer<org.apache.pulsar.client.api.schema.GenericRecord> consumer, org.apache.pulsar.client.api.Message<org.apache.pulsar.client.api.schema.GenericRecord> msg, boolean async) throws org.apache.pulsar.client.api.PulsarClientException
      Throws:
      org.apache.pulsar.client.api.PulsarClientException

    • handleFailures

      private void handleFailures(org.apache.nifi.processor.ProcessSession session, BlockingQueue<org.apache.pulsar.client.api.Message<org.apache.pulsar.client.api.schema.GenericRecord>> parseFailures, byte[] demarcator)

    • handleAsync

      protected void handleAsync(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session, org.apache.pulsar.client.api.Consumer<org.apache.pulsar.client.api.schema.GenericRecord> consumer, org.apache.nifi.serialization.RecordReaderFactory readerFactory, org.apache.nifi.serialization.RecordSetWriterFactory writerFactory, byte[] demarcator) throws org.apache.pulsar.client.api.PulsarClientException
      Pull messages off of the CompletableFuture's held in the consumerService and process them in a batch.
      Parameters:
      context - - The current ProcessContext
      session - - The current ProcessSession.
      consumer - - The Pulsar consumer.
      readerFactory - - The factory used to read the messages.
      writerFactory - - The factory used to write the messages.
      demarcator - - The bytes used to demarcate the individual messages.
      Throws:
      org.apache.pulsar.client.api.PulsarClientException - if there is an issue connecting to the Pulsar cluster.

    • getSchema

      private org.apache.nifi.serialization.record.RecordSchema getSchema(org.apache.nifi.flowfile.FlowFile flowFile, org.apache.nifi.serialization.RecordReaderFactory readerFactory, byte[] msgValue)

    • getRecordWriter

      private org.apache.nifi.serialization.RecordSetWriter getRecordWriter(org.apache.nifi.serialization.RecordSetWriterFactory writerFactory, org.apache.nifi.serialization.record.RecordSchema srcSchema, OutputStream out, org.apache.nifi.flowfile.FlowFile flowFile)