Package org.apache.pinot.plugin.segmentwriter.filebased

Class FileBasedSegmentWriter

java.lang.Object

org.apache.pinot.plugin.segmentwriter.filebased.FileBasedSegmentWriter

All Implemented Interfaces:

Closeable, AutoCloseable, SegmentWriter

@NotThreadSafe
public class FileBasedSegmentWriter
extends Object
implements SegmentWriter

A SegmentWriter implementation that uses a local file as a buffer to collect GenericRow. The GenericRow are written to the buffer as AVRO records.

Constructor Summary

Constructors

Constructor	Description
FileBasedSegmentWriter()

Method Summary

Modifier and Type	Method	Description
void	close()
void	collect(GenericRow row)
URI	flush()	Creates one Pinot segment using the GenericRows collected in the AVRO file buffer, at the outputDirUri as specified in the tableConfig->batchConfigs.
void	init(TableConfig tableConfig, Schema schema)
void	init(TableConfig tableConfig, Schema schema, Map<String,String> batchConfigOverride)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pinot.spi.ingestion.segment.writer.SegmentWriter

collect

Constructor Detail

FileBasedSegmentWriter

public FileBasedSegmentWriter()

Method Detail

init

public void init(TableConfig tableConfig,
                 Schema schema)
          throws Exception

Specified by:

init in interface SegmentWriter

Throws:

Exception

init

public void init(TableConfig tableConfig,
                 Schema schema,
                 Map<String,String> batchConfigOverride)
          throws Exception

Specified by:

init in interface SegmentWriter

Throws:

Exception

collect

public void collect(GenericRow row)
             throws IOException

Specified by:

collect in interface SegmentWriter

Throws:

IOException

flush

public URI flush()
          throws IOException

Creates one Pinot segment using the GenericRows collected in the AVRO file buffer, at the outputDirUri as specified in the tableConfig->batchConfigs. Successful invocation of this method means that the GenericRows collected so far, are now available in the Pinot segment and not available in the buffer anymore. Successful completion of segment will return the segment URI. The buffer will be reset and ready to accept further records via collect() If an exception is thrown, the buffer will not be reset and so, flush() can be invoked repeatedly in a retry loop. If a successful invocation is not achieved,close() followed by init will have to be called in order to reset the buffer and resume record writing.

Specified by:

flush in interface SegmentWriter

Returns:

URI of the generated segment

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException