Package net.morimekta.providence.serializer.pretty

Class Tokenizer

java.lang.Object

java.io.Reader

net.morimekta.util.io.LineBufferedReader

net.morimekta.providence.serializer.pretty.Tokenizer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.lang.Readable

public class Tokenizer
extends net.morimekta.util.io.LineBufferedReader

Simple tokenizer for the pretty serializer that strips away comments based on the "#" (shell) comment character. Each comment lasts until the next newline.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	Tokenizer.TokenValidator

Field Summary

Fields

Modifier and Type	Field	Description
static int	DEFAULT_BUFFER_SIZE

Fields inherited from class net.morimekta.util.io.LineBufferedReader

buffer, bufferLimit, bufferLineEnd, bufferOffset, lastChar, lineNo, linePos, preLoaded, reader

Fields inherited from class java.io.Reader

lock

Constructor Summary

Constructors

Constructor	Description
Tokenizer(java.io.InputStream in)	Create a JSON tokenizer that reads from the input steam.
Tokenizer(java.io.InputStream in, int bufferSize)	Create a JSON tokenizer that reads from the input steam.
Tokenizer(java.io.Reader in, int bufferSize, boolean preLoadAll)	Create a tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.

Method Summary

Modifier and Type	Method	Description
protected TokenizerException	eof(java.lang.String format, java.lang.Object... params)
Token	expect(java.lang.String expected)	Expect a new JSON token on the stream.
Token	expect(java.lang.String expected, Tokenizer.TokenValidator validator)	Expect at a valid token containing anything.
Token	expectIdentifier(java.lang.String message)
Token	expectInteger(java.lang.String message)
Token	expectLiteral(java.lang.String message)
char	expectSymbol(java.lang.String expected, char... symbols)
protected TokenizerException	failure(int startLineNo, int startLinePos, int length, java.lang.String format, java.lang.Object... params)
protected TokenizerException	failure(java.lang.String format, java.lang.Object... params)
TokenizerException	failure(Token token, java.lang.String message, java.lang.Object... params)
Token	getLastToken()
boolean	hasNext()	Whether there is another token on the stream.
Token	next()	Returns the next token on the stream, or null if there are no more JSON tokens on the stream.
protected Token	nextSymbol()
Token	peek()	Return the next token or throw an exception.
Token	peek(java.lang.String message)	Return the next token or throw an exception.
java.lang.String	readBinary(char end)	Read the 'content' of encoded binary.

Methods inherited from class net.morimekta.util.io.LineBufferedReader

close, getLine, getLineNo, getLinePos, getRemainingLines, getRestOfLine, maybeConsolidateBuffer, read, read, readNextChar

Methods inherited from class java.io.Reader

mark, markSupported, nullReader, read, read, ready, reset, skip, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BUFFER_SIZE

public static final int DEFAULT_BUFFER_SIZE

See Also:

Constant Field Values

Constructor Detail

Tokenizer

public Tokenizer(java.io.InputStream in)

Create a JSON tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.

Parameters:

in - Input stream to parse from.

Tokenizer

public Tokenizer(java.io.InputStream in,
                 int bufferSize)

Create a JSON tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.

Parameters:

in - Input stream to parse from.

bufferSize - The size of the char buffer. Default is 2048 chars (4096 bytes).

Tokenizer

public Tokenizer(java.io.Reader in,
                 int bufferSize,
                 boolean preLoadAll)

Create a tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.

Parameters:

in - Reader of content to parse.

bufferSize - The size of the char buffer. Default is 2048 chars (4096 bytes).

preLoadAll - Load all content up front. Handy for config and thrift program files.

Method Detail

expect

@Nonnull
public Token expect(@Nonnull
                    java.lang.String expected)
             throws java.io.IOException

Expect a new JSON token on the stream.

Parameters:

expected - Message to add to exception if there are no more JSON tokens on the stream.

Returns:

The next token.

Throws:

java.io.IOException - If unable to read from stream.

expect

public Token expect(@Nonnull
                    java.lang.String expected,
                    @Nonnull
                    Tokenizer.TokenValidator validator)
             throws java.io.IOException

Expect at a valid token containing anything.

Parameters:

expected - The expectation description.

validator - Validator callback.

Returns:

The token.

Throws:

java.io.IOException - If failed to read a token.

expectSymbol

public char expectSymbol(@Nonnull
                         java.lang.String expected,
                         char... symbols)
                  throws java.io.IOException

Parameters:

expected - Message to add to exception if there are no more JSON tokens on the stream.

symbols - List of symbol characters to expect.

Returns:

The symbol that was encountered.

Throws:

java.io.IOException - If unable to read from stream.

expectIdentifier

public Token expectIdentifier(@Nonnull
                              java.lang.String message)
                       throws java.io.IOException

Throws:

java.io.IOException

expectInteger

@Nonnull
public Token expectInteger(java.lang.String message)
                    throws java.io.IOException

Throws:

java.io.IOException

expectLiteral

@Nonnull
public Token expectLiteral(java.lang.String message)
                    throws java.io.IOException

Throws:

java.io.IOException

hasNext

public boolean hasNext()
                throws java.io.IOException

Whether there is another token on the stream. This will read up until it finds a JSON token, or until the stream ends.

Returns:

True if (and only if) there is at least one more token on the stream.

Throws:

java.io.IOException - If unable to read from stream.

getLastToken

public Token getLastToken()

readBinary

public java.lang.String readBinary(char end)
                            throws java.io.IOException

Read the 'content' of encoded binary. This does not parse the binary, just read out from the buffer the string representing the binary data, as delimited by the requested 'end' char.

Parameters:

end - The char that ends the binary content.

Returns:

The string encoded string representation.

Throws:

TokenizerException - On illegal content.

java.io.IOException

peek

public Token peek()
           throws java.io.IOException

Return the next token or throw an exception. Though it does not consume that token.

Returns:

The next token.

Throws:

java.io.IOException - If unable to read from stream.

peek

@Nonnull
public Token peek(java.lang.String message)
           throws java.io.IOException

Return the next token or throw an exception. Though it does not consume that token.

Parameters:

message - Message to add to exception if there are no more JSON tokens on the stream.

Returns:

The next token.

Throws:

java.io.IOException - If unable to read from stream.

next

@Nullable
public Token next()
           throws java.io.IOException

Returns the next token on the stream, or null if there are no more JSON tokens on the stream.

Returns:

The next token, or null.

Throws:

java.io.IOException - If unable to read from stream.

nextSymbol

@Nonnull
protected Token nextSymbol()
                    throws java.io.IOException

Throws:

java.io.IOException

failure

@Nonnull
public TokenizerException failure(Token token,
                                  java.lang.String message,
                                  java.lang.Object... params)

failure

@Nonnull
protected final TokenizerException failure(int startLineNo,
                                           int startLinePos,
                                           int length,
                                           java.lang.String format,
                                           java.lang.Object... params)

eof

@Nonnull
protected final TokenizerException eof(java.lang.String format,
                                       java.lang.Object... params)

failure

@Nonnull
protected TokenizerException failure(java.lang.String format,
                                     java.lang.Object... params)