net.morimekta.providence.serializer.pretty

Class PrettyTokenizer

java.lang.Object

java.io.Reader

net.morimekta.util.io.LineBufferedReader

net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

net.morimekta.providence.serializer.pretty.PrettyTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable, Readable, net.morimekta.util.lexer.Tokenizer<PrettyTokenType,PrettyToken>

public class PrettyTokenizer
extends net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

Simple tokenizer for the pretty serializer that strips away comments based on the "#" (shell) comment character. Each comment lasts until the next newline.

Field Summary

Fields inherited from class net.morimekta.util.lexer.TokenizerBase

DEFAULT_BUFFER_SIZE

Fields inherited from class net.morimekta.util.io.LineBufferedReader

buffer, bufferLimit, bufferLineEnd, bufferOffset, lastChar, lineNo, linePos, preLoaded, reader

Fields inherited from class java.io.Reader

lock

Constructor Summary

Constructors

Constructor and Description
PrettyTokenizer(InputStream in) Create a pretty tokenizer that reads from the input steam.
PrettyTokenizer(InputStream in, int bufferSize) Create a pretty tokenizer that reads from the input steam.
PrettyTokenizer(Reader in, int bufferSize, boolean preLoadAll) Create a pretty tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.

Method Summary

Modifier and Type	Method and Description
protected PrettyToken	genericToken(char[] buffer, int offset, int len, PrettyTokenType type, int lineNo, int linePos)
protected PrettyToken	identifierToken(char[] buffer, int offset, int len, int lineNo, int linePos)
protected PrettyToken	numberToken(char[] buffer, int offset, int len, int lineNo, int linePos)
protected boolean	startString()
protected PrettyToken	stringToken(char[] buffer, int offset, int len, int lineNo, int linePos)
protected PrettyToken	symbolToken(char[] buffer, int offset, int len, int lineNo, int linePos)

Methods inherited from class net.morimekta.util.lexer.TokenizerBase

allowIdentifier, currentLine, currentLineNo, currentLinePos, eofFailure, failure, identifierSeparator, isWhitespace, nextSymbol, parseNextToken, readUntil, startIdentifier, startNumber, startSymbol, toString

Methods inherited from class net.morimekta.util.io.LineBufferedReader

close, getLine, getLineNo, getLinePos, getRemainingLines, getRestOfLine, maybeConsolidateBuffer, read, read, readNextChar

Methods inherited from class java.io.Reader

mark, markSupported, read, read, ready, reset, skip

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PrettyTokenizer

public PrettyTokenizer(InputStream in)

Create a pretty tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.

Parameters:

in - Input stream to parse from.

PrettyTokenizer

public PrettyTokenizer(InputStream in,
                       int bufferSize)

Create a pretty tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.

Parameters:

in - Input stream to parse from.

bufferSize - The size of the char buffer. Default is 2048 chars (4096 bytes).

PrettyTokenizer

public PrettyTokenizer(Reader in,
                       int bufferSize,
                       boolean preLoadAll)

Create a pretty tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.

Parameters:

in - Reader of content to parse.

bufferSize - The size of the char buffer. Default is 2048 chars (4096 bytes).

preLoadAll - Load all content up front. Handy for config and thrift program files.

Method Detail

startString

protected boolean startString()

Overrides:

startString in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

genericToken

protected PrettyToken genericToken(char[] buffer,
                                   int offset,
                                   int len,
                                   @Nonnull
                                   PrettyTokenType type,
                                   int lineNo,
                                   int linePos)

Specified by:

genericToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

identifierToken

protected PrettyToken identifierToken(char[] buffer,
                                      int offset,
                                      int len,
                                      int lineNo,
                                      int linePos)

Specified by:

identifierToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

stringToken

protected PrettyToken stringToken(char[] buffer,
                                  int offset,
                                  int len,
                                  int lineNo,
                                  int linePos)

Specified by:

stringToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

numberToken

protected PrettyToken numberToken(char[] buffer,
                                  int offset,
                                  int len,
                                  int lineNo,
                                  int linePos)

Specified by:

numberToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>

symbolToken

protected PrettyToken symbolToken(char[] buffer,
                                  int offset,
                                  int len,
                                  int lineNo,
                                  int linePos)

Specified by:

symbolToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,PrettyToken>