com.twitter.distributedlog

Class LogRecord

java.lang.Object

com.twitter.distributedlog.LogRecord

Direct Known Subclasses:

LogRecordWithDLSN

public class LogRecord
extends Object

Log record is the basic element in a log.

A log is a sequence of log records. Each log record is a sequence of bytes. Log records are written sequentially into a stream, and will be assigned with an unique system generated sequence number DLSN (distributedlog sequence number). Besides DLSN, application can assign its own sequence number while constructing log records. The application defined sequence number is called TransactionID (txid). Either DLSN or TransactionId could be used to position readers to start from specific log records.

User Record

User records are the records written by applications and read by applications. They are constructed via LogRecord(long, byte[]) by applications and appended to logs by writers. And they would be deserialized from bytes by the readers and return to applications.

Control Record

Control records are special records that written by distributedlog. They are invisible to applications. They could be treated as commit requests as what people could find in distributed consensus algorithms, since they are usually written by distributedlog to commit application written records. Commit means making application written records visible to readers to achieve consistent views among them.

They are named as 'Control Records' for controlling visibility of application written records.

The transaction id of 'Control Records' are assigned by distributedlog by inheriting from last written user records. So we could indicate what user records that a control record is committing by looking at its transaction id.

EndOfStream Record

EoS(EndOfStream) is a special control record that would be written by a writer to seal a log. After a EoS record is written to a log, no writers could append any record after that and readers will get EndOfStreamException when they reach EoS.

TransactionID of EoS is Long.MAX_VALUE.

Serialization & Deserialization

Data type in brackets. Interpretation should be on the basis of data types and not individual bytes to honor Endianness.

 LogRecord structure:
 -------------------
 Bytes 0 - 7                      : Metadata (Long)
 Bytes 8 - 15                     : TxId (Long)
 Bytes 16 - 19                    : Payload length (Integer)
 Bytes 20 - 20+payload.length-1   : Payload (Byte[])

 Metadata: 8 Bytes (Long)
 --------

 0x 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            |_____________| |_____|
                   |           |
               position      flags

 Flags: 2 Bytes (least significant)
 -----
 Bit  0      : If set, control record, else record with payload.
 Bit  1      : If set, end of stream.
 Bits 2 - 15 : Unused
 

Sequence Numbers

A record is associated with three types of sequence numbers. They are generated and used for different purposes. Check LogRecordWithDLSN for more details.

See Also:

LogRecordWithDLSN

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LogRecord.Reader This class is a package private class for reading log records from the persistent
static class	LogRecord.Writer Class for writing log records

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_LOGRECORD_SIZE
static int	MAX_LOGRECORDSET_SIZE

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	LogRecord() Construct an uninitialized log record.
	LogRecord(long txid, byte[] payload) Construct a log record with TransactionId and payload.

Method Summary

Methods

Modifier and Type	Method and Description
protected long	getMetadata()
byte[]	getPayload() Return the payload of this log record.
InputStream	getPayLoadInputStream() Return the payload as an InputStream.
int	getPositionWithinLogSegment() The position in the log segment means how many records (inclusive) added to the log segment so far.
long	getTransactionId() Return application defined transaction id.
boolean	isControl() Check if the record is a control record.
static boolean	isControl(long flags) Check flags to see if it indicates a control record.
boolean	isRecordSet() Check if the record represents a set of records.
static boolean	isRecordSet(long metadata)
protected void	readPayload(DataInputStream in)
void	setControl()
protected void	setMetadata(long metadata)
void	setRecordSet() Set the record to represent a set of records.
protected void	setTransactionId(long txid) Set application defined transaction id.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_LOGRECORD_SIZE

public static final int MAX_LOGRECORD_SIZE

See Also:

Constant Field Values

MAX_LOGRECORDSET_SIZE

public static final int MAX_LOGRECORDSET_SIZE

See Also:

Constant Field Values

Constructor Detail

LogRecord

protected LogRecord()

Construct an uninitialized log record.

NOTE: only deserializer should call this constructor.

LogRecord

public LogRecord(long txid,
         byte[] payload)

Construct a log record with TransactionId and payload.

Usually writer would construct the log record for writing.

Parameters:

txid - application defined transaction id.

payload - record data

Method Detail

getTransactionId

public long getTransactionId()

Return application defined transaction id.

Returns:

transacton id.

setTransactionId

protected void setTransactionId(long txid)

Set application defined transaction id.

Parameters:

txid - application defined transaction id.

getPayload

public byte[] getPayload()

Return the payload of this log record.

Returns:

payload of this log record.

getPayLoadInputStream

public InputStream getPayLoadInputStream()

Return the payload as an InputStream.

Returns:

payload as input stream

setMetadata

protected void setMetadata(long metadata)

getMetadata

protected long getMetadata()

getPositionWithinLogSegment

public int getPositionWithinLogSegment()

The position in the log segment means how many records (inclusive) added to the log segment so far.

Returns:

position of the record in the log segment.

setRecordSet

public void setRecordSet()

Set the record to represent a set of records.

The bytes in this record is the serialized format of LogRecordSet.

isRecordSet

public boolean isRecordSet()

Check if the record represents a set of records.

Returns:

true if the record represents a set of records, otherwise false.

See Also:

setRecordSet()

isRecordSet

public static boolean isRecordSet(long metadata)

setControl

public void setControl()

isControl

public boolean isControl()

Check if the record is a control record.

Returns:

true if the record is a control record, otherwise false.

isControl

public static boolean isControl(long flags)

Check flags to see if it indicates a control record.

Parameters:

flags - record flags

Returns:

true if the record is a control record, otherwise false.

readPayload

protected void readPayload(DataInputStream in)
                    throws IOException

Throws:

IOException