Package org.apache.druid.indexing.overlord.supervisor

Interface StreamSupervisor

All Superinterfaces:
Supervisor
public interface StreamSupervisor extends Supervisor
An interface for managing supervisors that handle stream-based ingestion tasks.

This interface extends Supervisor and adds additional functionality for managing the lifecycle of stream-based ingestion tasks.

  • Method Details

    • resetOffsets

      void resetOffsets(DataSourceMetadata resetDataSourceMetadata)
      Reset offsets with provided dataSource metadata. The resulting stored offsets should be a union of existing checkpointed offsets with provided offsets.
      Parameters:
      resetDataSourceMetadata - required datasource metadata with offsets to reset.
      Throws:
      DruidException - if any metadata attribute doesn't match the supervisor's state.

    • checkpoint

      void checkpoint(int taskGroupId, DataSourceMetadata checkpointMetadata)
      The definition of checkpoint is not very strict as currently it does not affect data or control path. On this call Supervisor can potentially checkpoint data processed so far to some durable storage for example - Kafka/Kinesis Supervisor uses this to merge and handoff segments containing at least the data represented by DataSourceMetadata
      Parameters:
      taskGroupId - unique Identifier to figure out for which sequence to do checkpointing
      checkpointMetadata - metadata for the sequence to currently checkpoint

    • computeLagStats

      LagStats computeLagStats()
      Computes maxLag, totalLag and avgLag

    • getActiveTaskGroupsCount

      int getActiveTaskGroupsCount()

    • handoffTaskGroupsEarly

      default void handoffTaskGroupsEarly(List<Integer> taskGroupIds)
      Marks the given task groups as ready for segment hand-off irrespective of the task run times. In the subsequent run, the supervisor initiates segment publish and hand-off for these task groups and rolls over their tasks. taskGroupIds that are not valid or not actively reading are simply ignored.