Package org.apache.druid.server.coordinator.duty

Class KillUnreferencedSegmentSchema

java.lang.Object
org.apache.druid.server.coordinator.duty.MetadataCleanupDuty
org.apache.druid.server.coordinator.duty.KillUnreferencedSegmentSchema
All Implemented Interfaces:
CoordinatorDuty
public class KillUnreferencedSegmentSchema extends MetadataCleanupDuty
Coordinator duty to clean up segment schema which are not referenced by any used segment.

  1. If a schema is not referenced, UPDATE schemas SET used = false, used_status_last_updated = now
  2. DELETE FROM schemas WHERE used = false AND used_status_last_updated < 6 hours ago
  3. When creating a new segment, try to find schema for the fingerprint of the segment.
    1. If no record found, create a new one.
    2. If record found which has used = true, reuse this schema_id.
    3. If record found which has used = false, UPDATE SET used = true, used_status_last_updated = now

Possible race conditions:

  1. Between ops 1 and 3b: In other words, we might end up with a segment that points to a schema that has just been marked as unused. This is repaired by the coordinator duty.
  2. Between 2 and 3c: This can be handled. Either 2 will fail to update any rows (good case) or 3c will fail to update any rows (bad case). In the bad case, we need to recreate the schema, same as step 3a.

  • Constructor Details

  • Method Details

    • cleanupEntriesCreatedBefore

      protected int cleanupEntriesCreatedBefore(org.joda.time.DateTime minCreatedTime)
      Description copied from class: MetadataCleanupDuty
      Cleans up metadata entries created before the minCreatedTime.

      This method is not invoked if the cleanupPeriod has not elapsed since the MetadataCleanupDuty.lastCleanupTime.

      Specified by:
      cleanupEntriesCreatedBefore in class MetadataCleanupDuty
      Returns:
      Number of deleted metadata entries