uk.co.cogitolearning.cogpar

Class Tokenizer

java.lang.Object

uk.co.cogitolearning.cogpar.Tokenizer

public class Tokenizer
extends Object

A class for reading an input string and separating it into tokens that can be fed into Parser. The user can add regular expressions that will be matched against the front of the string. Regular expressions should not contain beginning-of-string or end-of-string anchors or any capturing groups as these will be added by the tokenizer itself.

Constructor Summary

Constructors

Constructor and Description
Tokenizer() Default constructor

Method Summary

Modifier and Type	Method and Description
void	add(String regex, int token) Add a regular expression and a token id to the internal list of recognized tokens
static Tokenizer	getExpressionTokenizer() A static method that returns a tokenizer for mathematical expressions
LinkedList<Token>	getTokens() Get the tokens generated in the last call to tokenize.
void	tokenize(String string) Tokenize an input string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer()

Default constructor

Method Detail

getExpressionTokenizer

public static Tokenizer getExpressionTokenizer()

A static method that returns a tokenizer for mathematical expressions

Returns:

a tokenizer that can handle mathematical expressions

add

public void add(String regex,
                int token)

Add a regular expression and a token id to the internal list of recognized tokens

Parameters:

regex - the regular expression to match against

token - the token id that the regular expression is linked to

tokenize

public void tokenize(String string)

Tokenize an input string. The result of tokenizing can be accessed via getTokens

Parameters:

string - the string to tokenize

getTokens

public LinkedList<Token> getTokens()

Get the tokens generated in the last call to tokenize.

Returns:

a list of tokens to be fed to Parser