Package org.apache.iotdb.db.engine.storagegroup

Class DataRegion

  • public class DataRegion
extends java.lang.Object
    For sequence data, a DataRegion has some TsFileProcessors, in which there is only one TsFileProcessor in the working status.

    There are two situations to set the working TsFileProcessor to closing status:

    (1) when inserting data into the TsFileProcessor, and the TsFileProcessor shouldFlush() (or shouldClose())

    (2) someone calls syncCloseAllWorkingTsFileProcessors(). (up to now, only flush command from cli will call this method)

    UnSequence data has the similar process as above.

    When a sequence TsFileProcessor is submitted to be flushed, the updateLatestFlushTimeCallback() method will be called as a callback.

    When a TsFileProcessor is closed, the closeUnsealedTsFileProcessorCallBack() method will be called as a callback.

    • Field Detail

      • COMPACTION_TASK_SUBMIT_DELAY

        public static final long COMPACTION_TASK_SUBMIT_DELAY
        See Also:
        Constant Field Values

    • Constructor Detail

      • DataRegion

        public DataRegion​(java.lang.String systemDir,
                  java.lang.String dataRegionId,
                  TsFileFlushPolicy fileFlushPolicy,
                  java.lang.String storageGroupName)
           throws DataRegionException
        constrcut a storage group processor
        Parameters:
        systemDir - system dir path
        dataRegionId - data region id e.g. 1
        fileFlushPolicy - file flush policy
        storageGroupName - logical storage group name e.g. root.sg1
        Throws:
        DataRegionException

      • DataRegion

        public DataRegion​(java.lang.String storageGroupName,
                  java.lang.String id)

    • Method Detail

      • getStorageGroupName

        public java.lang.String getStorageGroupName()

      • isReady

        public boolean isReady()

      • setReady

        public void setReady​(boolean ready)

      • getIsSettling

        public java.util.concurrent.atomic.AtomicBoolean getIsSettling()

      • setSettling

        public void setSettling​(boolean isSettling)

      • submitAFlushTask

        public boolean submitAFlushTask​(long timeRangeId,
                                boolean sequence,
                                IMemTable memTable)
        WAL module uses this method to flush memTable
        Returns:
        True if flush task is submitted successfully

      • submitAFlushTaskWhenShouldFlush

        public void submitAFlushTaskWhenShouldFlush​(TsFileProcessor tsFileProcessor)
        mem control module uses this method to flush memTable
        Parameters:
        tsFileProcessor - tsfile processor in which memTable to be flushed

      • syncCloseOneTsFileProcessor

        public void syncCloseOneTsFileProcessor​(boolean sequence,
                                        TsFileProcessor tsFileProcessor)
        close one tsfile processor
        Parameters:
        sequence - whether this tsfile processor is sequence or not
        tsFileProcessor - tsfile processor

      • asyncCloseOneTsFileProcessor

        public void asyncCloseOneTsFileProcessor​(boolean sequence,
                                         TsFileProcessor tsFileProcessor)
        close one tsfile processor, thread-safety should be ensured by caller
        Parameters:
        sequence - whether this tsfile processor is sequence or not
        tsFileProcessor - tsfile processor

      • deleteFolder

        public void deleteFolder​(java.lang.String systemDir)
        delete the storageGroup's own folder in folder data/system/storage_groups
        Parameters:
        systemDir - system dir

      • closeAllResources

        public void closeAllResources()
        close all tsfile resource

      • syncDeleteDataFiles

        public void syncDeleteDataFiles()
        delete tsfile

      • checkFilesTTL

        public void checkFilesTTL()
        Iterate each TsFile and try to lock and remove those out of TTL.

      • timedFlushSeqMemTable

        public void timedFlushSeqMemTable()

      • timedFlushUnseqMemTable

        public void timedFlushUnseqMemTable()

      • syncCloseAllWorkingTsFileProcessors

        public void syncCloseAllWorkingTsFileProcessors()
        This method will be blocked until all tsfile processors are closed.

      • asyncCloseAllWorkingTsFileProcessors

        public void asyncCloseAllWorkingTsFileProcessors()
        close all working tsfile processors

      • query

        public QueryDataSource query​(java.util.List<org.apache.iotdb.commons.path.PartialPath> pathList,
                             java.lang.String singleDeviceId,
                             QueryContext context,
                             QueryFileManager filePathsManager,
                             org.apache.iotdb.tsfile.read.filter.basic.Filter timeFilter)
                      throws QueryProcessException
        build query data source by searching all tsfile which fit in query filter
        Parameters:
        pathList - data paths
        context - query context
        timeFilter - time filter
        singleDeviceId - selected deviceId (not null only when all the selected series are under the same device)
        Returns:
        query data source
        Throws:
        QueryProcessException

      • readLock

        public void readLock()
        lock the read lock of the insert lock

      • readUnlock

        public void readUnlock()
        unlock the read lock of insert lock

      • writeLock

        public void writeLock​(java.lang.String holder)
        lock the write lock of the insert lock

      • writeUnlock

        public void writeUnlock()
        unlock the write lock of the insert lock

      • delete

        public void delete​(org.apache.iotdb.commons.path.PartialPath path,
                   long startTime,
                   long endTime,
                   long planIndex,
                   DataRegion.TimePartitionFilter timePartitionFilter)
            throws java.io.IOException
        Delete data whose timestamp <= 'timestamp' and belongs to the time series deviceId.measurementId.
        Parameters:
        path - the timeseries path of the to be deleted.
        startTime - the startTime of delete range.
        endTime - the endTime of delete range.
        timePartitionFilter -
        Throws:
        java.io.IOException

      • deleteByDevice

        public void deleteByDevice​(org.apache.iotdb.commons.path.PartialPath pattern,
                           long startTime,
                           long endTime,
                           long searchIndex,
                           DataRegion.TimePartitionFilter timePartitionFilter)
                    throws java.io.IOException
        Parameters:
        pattern - Must be a pattern start with a precise device path
        startTime -
        endTime -
        searchIndex -
        timePartitionFilter -
        Throws:
        java.io.IOException

      • updateNewlyFlushedPartitionLatestFlushedTimeForEachDevice

        public void updateNewlyFlushedPartitionLatestFlushedTimeForEachDevice​(long partitionId,
                                                                      java.lang.String deviceId,
                                                                      long time)
        used for upgrading

      • countUpgradeFiles

        public int countUpgradeFiles()
        count all Tsfiles in the storage group which need to be upgraded
        Returns:
        total num of the tsfiles which need to be upgraded in the storage group

      • upgrade

        public void upgrade()
        upgrade all files belongs to this storage group

      • compact

        public void compact()
        merge file under this storage group processor

      • loadNewTsFile

        public void loadNewTsFile​(TsFileResource newTsFileResource,
                          boolean deleteOriginFile)
                   throws LoadFileException
        Load a new tsfile to storage group processor. Tne file may have overlap with other files.

        that there has no file which is overlapping with the new file.

        Firstly, determine the loading type of the file, whether it needs to be loaded in sequence list or unsequence list.

        Secondly, execute the loading process by the type.

        Finally, update the latestTimeForEachDevice and partitionLatestFlushedTimeForEachDevice.

        Parameters:
        newTsFileResource - tsfile resource @UsedBy load external tsfile module
        deleteOriginFile - whether to delete origin tsfile
        Throws:
        LoadFileException

      • setPartitionFileVersionToMax

        public void setPartitionFileVersionToMax​(long partition,
                                         long version)
        Set the version in "partition" to "version" if "version" is larger than the current version.

      • removeFullyOverlapFiles

        public void removeFullyOverlapFiles​(TsFileResource resource)
        If the historical versions of a file is a sub-set of the given file's, (close and) remove it to reduce unnecessary merge. Only used when the file sender and the receiver share the same file close policy. Warning: DO NOT REMOVE

      • deleteTsfile

        public boolean deleteTsfile​(java.io.File tsfieToBeDeleted)
        Delete tsfile if it exists.

        Firstly, remove the TsFileResource from sequenceFileList/unSequenceFileList.

        Secondly, delete the tsfile and .resource file.

        Parameters:
        tsfieToBeDeleted - tsfile to be deleted
        Returns:
        whether the file to be deleted exists. @UsedBy sync module, load external tsfile module.

      • getWorkSequenceTsFileProcessors

        public java.util.Collection<TsFileProcessor> getWorkSequenceTsFileProcessors()
        get all working sequence tsfile processors
        Returns:
        all working sequence tsfile processors

      • unloadTsfile

        public boolean unloadTsfile​(java.io.File fileToBeUnloaded,
                            java.io.File targetDir)
        Unload tsfile and move it to the target directory if it exists.

        Firstly, unload the TsFileResource from sequenceFileList/unSequenceFileList.

        Secondly, move the tsfile and .resource file to the target directory.

        Parameters:
        fileToBeUnloaded - tsfile to be unloaded
        Returns:
        whether the file to be unloaded exists. @UsedBy load external tsfile module.

      • getWorkUnsequenceTsFileProcessors

        public java.util.Collection<TsFileProcessor> getWorkUnsequenceTsFileProcessors()
        get all working unsequence tsfile processors
        Returns:
        all working unsequence tsfile processors

      • setDataTTL

        public void setDataTTL​(long dataTTL)

      • getSequenceFileList

        public java.util.List<TsFileResource> getSequenceFileList()

      • getUnSequenceFileList

        public java.util.List<TsFileResource> getUnSequenceFileList()

      • getDataRegionId

        public java.lang.String getDataRegionId()

      • getStorageGroupPath

        public java.lang.String getStorageGroupPath()
        Get the storageGroupPath with dataRegionId.
        Returns:
        data region path, like root.sg1/0

      • isFileAlreadyExist

        public boolean isFileAlreadyExist​(TsFileResource tsFileResource,
                                  long partitionNum)
        Check if the data of "tsFileResource" all exist locally by comparing planIndexes in the partition of "partitionNumber". This is available only when the IoTDB instances which generated "tsFileResource" have the same plan indexes as the local one.
        Returns:
        true if any file contains plans with indexes no less than the max plan index of "tsFileResource", otherwise false.

      • abortCompaction

        public void abortCompaction()

      • getTsFileResourceManager

        public TsFileManager getTsFileResourceManager()

      • getPartitionMaxFileVersions

        public long getPartitionMaxFileVersions​(long partitionId)

      • addSettleFilesToList

        public void addSettleFilesToList​(java.util.List<TsFileResource> seqResourcesToBeSettled,
                                 java.util.List<TsFileResource> unseqResourcesToBeSettled,
                                 java.util.List<java.lang.String> tsFilePaths)

      • collectHistoryTsFileForSync

        public java.util.List<java.io.File> collectHistoryTsFileForSync​(ISyncManager syncManager,
                                                                long dataStartTime)
        Used to collect history TsFiles(i.e. the tsfile whose memtable == null).
        Parameters:
        syncManager - ISyncManager which invokes to collect history TsFile
        dataStartTime - only collect history TsFiles which contains the data after the dataStartTime
        Returns:
        A list, which contains TsFile path

      • setCustomCloseFileListeners

        public void setCustomCloseFileListeners​(java.util.List<CloseFileListener> customCloseFileListeners)

      • setCustomFlushListeners

        public void setCustomFlushListeners​(java.util.List<FlushListener> customFlushListeners)

      • setAllowCompaction

        public void setAllowCompaction​(boolean allowCompaction)

      • getTimePartitions

        public java.util.List<java.lang.Long> getTimePartitions()

      • getInsertWriteLockHolder

        public java.lang.String getInsertWriteLockHolder()

      • getTimedCompactionScheduleTask

        public java.util.concurrent.ScheduledExecutorService getTimedCompactionScheduleTask()

      • getIdTable

        public IDTable getIdTable()

      • getWALNode

        public IWALNode getWALNode()
        This method could only be used in multi-leader consensus

      • waitForDeleted

        public void waitForDeleted()
        Wait for this data region successfully deleted

      • markDeleted

        public void markDeleted()
        Release all threads waiting for this data region successfully deleted