Package org.apache.iotdb.db.tools

Class TsFileSplitByPartitionTool

  • All Implemented Interfaces:
    java.lang.AutoCloseable
    Direct Known Subclasses:
    TsFileOnlineUpgradeTool
    public class TsFileSplitByPartitionTool
extends java.lang.Object
implements java.lang.AutoCloseable

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected boolean checkIfNeedToDecode​(org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema, java.lang.String deviceId, org.apache.iotdb.tsfile.file.header.PageHeader pageHeader, long chunkHeaderOffset)
      If the page have no statistics or crosses multi partitions, will return true, otherwise return false.
      void close()  
      protected void decodeAndWritePage​(java.lang.String deviceId, org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema, java.nio.ByteBuffer pageData, java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl> partitionChunkWriterMap, long chunkHeaderOffset)  
      protected void endChunkGroup()  
      protected TsFileResource endFileAndGenerateResource​(org.apache.iotdb.tsfile.write.writer.TsFileIOWriter tsFileIOWriter)  
      protected boolean fileCheck()
      check if the file has correct magic strings and version number
      protected org.apache.iotdb.tsfile.write.writer.TsFileIOWriter getOrDefaultTsFileIOWriter​(java.io.File oldTsFile, long partition)  
      void parseAndRewriteFile​(java.util.List<TsFileResource> rewrittenResources)
      Parse the old files and generate some new files according to the time partition interval.
      protected void reWriteChunk​(java.lang.String deviceId, boolean firstChunkInChunkGroup, org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema, java.util.List<org.apache.iotdb.tsfile.file.header.PageHeader> pageHeadersInChunk, java.util.List<java.nio.ByteBuffer> pageDataInChunk, java.util.List<java.lang.Boolean> needToDecodeInfoInChunk, long chunkHeaderOffset)
      This method is for rewriting the Chunk which data is in the different time partitions.
      protected void rewritePageIntoFiles​(org.apache.iotdb.tsfile.read.common.BatchData batchData, org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema, java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl> partitionChunkWriterMap)  
      static void rewriteTsFile​(TsFileResource resourceToBeRewritten, java.util.List<TsFileResource> rewrittenResources)
      Rewrite an old file to the latest version
      java.lang.String upgradeTsFileName​(java.lang.String oldTsFileName)  
      protected void writePage​(org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema, org.apache.iotdb.tsfile.file.header.PageHeader pageHeader, java.nio.ByteBuffer pageData, java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl> partitionChunkWriterMap)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • reader

        protected org.apache.iotdb.tsfile.read.TsFileSequenceReader reader

      • oldTsFile

        protected java.io.File oldTsFile

      • oldModification

        protected java.util.List<Modification> oldModification

      • modsIterator

        protected java.util.Iterator<Modification> modsIterator

      • defaultTimeDecoder

        protected org.apache.iotdb.tsfile.encoding.decoder.Decoder defaultTimeDecoder

      • valueDecoder

        protected org.apache.iotdb.tsfile.encoding.decoder.Decoder valueDecoder

      • partitionWriterMap

        protected java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.writer.TsFileIOWriter> partitionWriterMap
        PartitionId -> TsFileIOWriter

      • maxPlanIndex

        protected long maxPlanIndex
        Maximum index of plans executed within this TsFile.

      • minPlanIndex

        protected long minPlanIndex
        Minimum index of plans executed within this TsFile.

    • Constructor Detail

      • TsFileSplitByPartitionTool

        public TsFileSplitByPartitionTool​(TsFileResource resourceToBeRewritten)
                           throws java.io.IOException
        Create a file reader of the given file. The reader will read the real data and rewrite to some new tsFiles.
        Throws:
        java.io.IOException - If some I/O error occurs

      • TsFileSplitByPartitionTool

        public TsFileSplitByPartitionTool​(TsFileResource resourceToBeRewritten,
                                  boolean needReaderForV2)
                           throws java.io.IOException
        Throws:
        java.io.IOException

    • Method Detail

      • rewriteTsFile

        public static void rewriteTsFile​(TsFileResource resourceToBeRewritten,
                                 java.util.List<TsFileResource> rewrittenResources)
                          throws java.io.IOException,
                                 org.apache.iotdb.tsfile.exception.write.WriteProcessException,
                                 org.apache.iotdb.commons.exception.IllegalPathException
        Rewrite an old file to the latest version
        Parameters:
        resourceToBeRewritten - the tsfile which to be rewrite
        rewrittenResources - the rewritten files
        Throws:
        java.io.IOException
        org.apache.iotdb.tsfile.exception.write.WriteProcessException
        org.apache.iotdb.commons.exception.IllegalPathException

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.io.IOException

      • parseAndRewriteFile

        public void parseAndRewriteFile​(java.util.List<TsFileResource> rewrittenResources)
                         throws java.io.IOException,
                                org.apache.iotdb.tsfile.exception.write.WriteProcessException,
                                org.apache.iotdb.commons.exception.IllegalPathException
        Parse the old files and generate some new files according to the time partition interval.
        Throws:
        java.io.IOException - WriteProcessException
        org.apache.iotdb.tsfile.exception.write.WriteProcessException
        org.apache.iotdb.commons.exception.IllegalPathException

      • checkIfNeedToDecode

        protected boolean checkIfNeedToDecode​(org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema,
                                      java.lang.String deviceId,
                                      org.apache.iotdb.tsfile.file.header.PageHeader pageHeader,
                                      long chunkHeaderOffset)
                               throws org.apache.iotdb.commons.exception.IllegalPathException
        If the page have no statistics or crosses multi partitions, will return true, otherwise return false.
        Throws:
        org.apache.iotdb.commons.exception.IllegalPathException

      • reWriteChunk

        protected void reWriteChunk​(java.lang.String deviceId,
                            boolean firstChunkInChunkGroup,
                            org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema,
                            java.util.List<org.apache.iotdb.tsfile.file.header.PageHeader> pageHeadersInChunk,
                            java.util.List<java.nio.ByteBuffer> pageDataInChunk,
                            java.util.List<java.lang.Boolean> needToDecodeInfoInChunk,
                            long chunkHeaderOffset)
                     throws java.io.IOException,
                            org.apache.iotdb.tsfile.exception.write.PageException,
                            org.apache.iotdb.commons.exception.IllegalPathException
        This method is for rewriting the Chunk which data is in the different time partitions. In this case, we have to decode the data to points, and then rewrite the data points to different chunkWriters, finally write chunks to their own upgraded TsFiles.
        Throws:
        java.io.IOException
        org.apache.iotdb.tsfile.exception.write.PageException
        org.apache.iotdb.commons.exception.IllegalPathException

      • endChunkGroup

        protected void endChunkGroup()
                      throws java.io.IOException
        Throws:
        java.io.IOException

      • upgradeTsFileName

        public java.lang.String upgradeTsFileName​(java.lang.String oldTsFileName)

      • getOrDefaultTsFileIOWriter

        protected org.apache.iotdb.tsfile.write.writer.TsFileIOWriter getOrDefaultTsFileIOWriter​(java.io.File oldTsFile,
                                                                                         long partition)

      • writePage

        protected void writePage​(org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema,
                         org.apache.iotdb.tsfile.file.header.PageHeader pageHeader,
                         java.nio.ByteBuffer pageData,
                         java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl> partitionChunkWriterMap)
                  throws org.apache.iotdb.tsfile.exception.write.PageException
        Throws:
        org.apache.iotdb.tsfile.exception.write.PageException

      • decodeAndWritePage

        protected void decodeAndWritePage​(java.lang.String deviceId,
                                  org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema,
                                  java.nio.ByteBuffer pageData,
                                  java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl> partitionChunkWriterMap,
                                  long chunkHeaderOffset)
                           throws java.io.IOException,
                                  org.apache.iotdb.commons.exception.IllegalPathException
        Throws:
        java.io.IOException
        org.apache.iotdb.commons.exception.IllegalPathException

      • rewritePageIntoFiles

        protected void rewritePageIntoFiles​(org.apache.iotdb.tsfile.read.common.BatchData batchData,
                                    org.apache.iotdb.tsfile.write.schema.MeasurementSchema schema,
                                    java.util.Map<java.lang.Long,​org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl> partitionChunkWriterMap)

      • fileCheck

        protected boolean fileCheck()
                     throws java.io.IOException
        check if the file has correct magic strings and version number
        Throws:
        java.io.IOException

      • endFileAndGenerateResource

        protected TsFileResource endFileAndGenerateResource​(org.apache.iotdb.tsfile.write.writer.TsFileIOWriter tsFileIOWriter)
                                             throws java.io.IOException
        Throws:
        java.io.IOException