org.apache.beam.sdk.util

Class BufferedElementCountingOutputStream

java.lang.Object

java.io.OutputStream

org.apache.beam.sdk.util.BufferedElementCountingOutputStream

All Implemented Interfaces:

java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable

@Internal
 @NotThreadSafe
public class BufferedElementCountingOutputStream
extends java.io.OutputStream

Provides an efficient encoding for Iterables containing small values by buffering up to bufferSize bytes of data before prefixing the count. Note that each element needs to be encoded in a nested context. See Coder.Context for more details.

To use this stream:

 BufferedElementCountingOutputStream os = ...
 for (Element E : elements) {
   os.markElementStart();
   // write an element to os
 }
 os.finish();
 

The resulting output stream is:

 countA element(0) element(1) ... element(countA - 1)
 countB element(0) element(1) ... element(countB - 1)
 ...
 countX element(0) element(1) ... element(countX - 1)
 countY
 

To read this stream:

 InputStream is = ...
 long count;
 do {
   count = VarInt.decodeLong(is);
   for (int i = 0; i < count; ++i) {
     // read an element from is
   }
 } while(count > 0);
 

The counts are encoded as variable length longs. See VarInt.encode(long, OutputStream) for more details. The end of the iterable is detected by reading a count of 0.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BUFFER_SIZE

Constructor Summary

Constructors

Constructor and Description
BufferedElementCountingOutputStream(java.io.OutputStream os) Creates an output stream which encodes the number of elements output to it in a streaming manner.
BufferedElementCountingOutputStream(java.io.OutputStream os, long terminatorValue)

Method Summary

Modifier and Type	Method and Description
void	close()
void	finish() Finishes the encoding by flushing any buffered data, and outputting a final count of 0.
void	flush()
void	markElementStart() Marks that a new element is being output.
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.io.OutputStream

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BUFFER_SIZE

public static final int DEFAULT_BUFFER_SIZE

See Also:

Constant Field Values

Constructor Detail

BufferedElementCountingOutputStream

public BufferedElementCountingOutputStream(java.io.OutputStream os)

Creates an output stream which encodes the number of elements output to it in a streaming manner.

BufferedElementCountingOutputStream

public BufferedElementCountingOutputStream(java.io.OutputStream os,
                                           long terminatorValue)

Method Detail

finish

public void finish()
            throws java.io.IOException

Finishes the encoding by flushing any buffered data, and outputting a final count of 0.

Throws:

java.io.IOException

markElementStart

public void markElementStart()
                      throws java.io.IOException

Marks that a new element is being output. This allows this output stream to use the buffer if it had previously overflowed marking the start of a new block of elements.

Throws:

java.io.IOException

write

public void write(int b)
           throws java.io.IOException

Specified by:

write in class java.io.OutputStream

Throws:

java.io.IOException

write

public void write(byte[] b,
                  int off,
                  int len)
           throws java.io.IOException

Overrides:

write in class java.io.OutputStream

Throws:

java.io.IOException

flush

public void flush()
           throws java.io.IOException

Specified by:

flush in interface java.io.Flushable

Overrides:

flush in class java.io.OutputStream

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class java.io.OutputStream

Throws:

java.io.IOException