org.openzen.zengarden.lexer

Class TokenStream<T extends Token<TT>,TT extends TokenType>

java.lang.Object

org.openzen.zengarden.lexer.TokenStream<T,TT>

Type Parameters:

T - token class

TT - token type class

All Implemented Interfaces:

java.util.Iterator<T>

public abstract class TokenStream<T extends Token<TT>,TT extends TokenType>
extends java.lang.Object
implements java.util.Iterator<T>

Represents a token stream. A token stream reads characters from a reader and presents it as a series of tokens. Can be used to implement LL(*) parsers. Token classes with a negative class are considered to be whitespace.

Constructor Summary

Constructors

Constructor and Description
TokenStream(java.lang.String filename, java.io.Reader reader, CompiledDFA<TT> dfa, TT eof) Creates a token stream using the specified reader and DFA.
TokenStream(java.lang.String filename, java.lang.String data, CompiledDFA<TT> dfa, TT eof) Creates a token stream which reads data from the specified string.

Method Summary

Modifier and Type	Method and Description
protected abstract T	createToken(TokenPosition position, java.lang.String value, TT tokenType) Called to create a token.
java.lang.String	getFile()
int	getLine()
int	getLineOffset()
TokenPosition	getPosition()
boolean	hasNext()
protected abstract T	invalidToken(TokenPosition position, java.lang.String token) Called when the input contains an invalid token.
protected abstract void	ioException(java.io.IOException ex) Called when an IO exception occurs.
boolean	isNext(TT type)
T	next()
T	optional(TT type)
T	peek()
void	popMark() Pops a mark from the mark stack without reset.
void	pushMark() Pushes a mark on the mark stack.
void	remove()
T	required(TT type, java.lang.String error)
protected abstract void	requiredTokenNotFound(TokenPosition position, java.lang.String error) Called when a required token could not be found.
void	reset() Pops a mark from the mark stack and resets the stream's position to it

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

TokenStream

public TokenStream(java.lang.String filename,
                   java.io.Reader reader,
                   CompiledDFA<TT> dfa,
                   TT eof)

Creates a token stream using the specified reader and DFA.

Parameters:

filename - filename

reader - reader to read characters from

dfa - DFA to tokenize the stream

eof - end of file token type

TokenStream

public TokenStream(java.lang.String filename,
                   java.lang.String data,
                   CompiledDFA<TT> dfa,
                   TT eof)

Creates a token stream which reads data from the specified string.

Parameters:

filename - filename

data - data to read

dfa - DFA to tokenize the stream

eof - end of file token type

Method Detail

getFile

public java.lang.String getFile()

getLine

public int getLine()

getLineOffset

public int getLineOffset()

peek

public T peek()

isNext

public boolean isNext(TT type)

optional

public T optional(TT type)

required

public T required(TT type,
                  java.lang.String error)

getPosition

public TokenPosition getPosition()

pushMark

public void pushMark()

Pushes a mark on the mark stack.

popMark

public void popMark()

Pops a mark from the mark stack without reset.

reset

public void reset()

Pops a mark from the mark stack and resets the stream's position to it

hasNext

public boolean hasNext()

Specified by:

hasNext in interface java.util.Iterator<T extends Token<TT>>

next

public T next()

Specified by:

next in interface java.util.Iterator<T extends Token<TT>>

remove

public void remove()

Specified by:

remove in interface java.util.Iterator<T extends Token<TT>>

createToken

protected abstract T createToken(TokenPosition position,
                                 java.lang.String value,
                                 TT tokenType)

Called to create a token. May also be used to postprocess tokens while generating them.

Parameters:

position - token position (range)

value - token value

tokenType - token type

Returns:

newly created token

requiredTokenNotFound

protected abstract void requiredTokenNotFound(TokenPosition position,
                                              java.lang.String error)

Called when a required token could not be found. Should log an error or throw an exception. If no exception is thrown, the calling required method will return null as token value.

Parameters:

position - erroring position

error - error to be logged

invalidToken

protected abstract T invalidToken(TokenPosition position,
                                  java.lang.String token)

Called when the input contains an invalid token. Should either create a token indicating an invalid token, or throw an exception.

Parameters:

position - erroring position

token - token value

Returns:

a token marking an invalid token

ioException

protected abstract void ioException(java.io.IOException ex)

Called when an IO exception occurs. Should throw an exception of some kind.

Parameters:

ex - exception to be logged