Sawyer::Lexer::Token< T > Class Template Reference

Description

template<typename T>
class Sawyer::Lexer::Token< T >

Represents one token of input.

Each token has a user-defined type which is some enumerated constant, or no type at all which means the token represents the end of the input stream. Tokens do not store their own lexeme, but rather point to the beginning and end of their lexeme in the input stream.

Definition at line 28 of file Lexer.h.

#include <Sawyer/Lexer.h>

Public Types
typedef T	TokenEnum

Public Member Functions
	Token ()
	Construct an EOF token.
	Token (TokenEnum type, size_t begin, size_t end)
	Construct a token.
bool	isEof () const
	Whether this is an EOF token.
TokenEnum	type () const
	Returns the token.
size_t	begin () const
	Token lexeme starting position.
size_t	end () const
	Token lexeme ending position.

Member Typedef Documentation

TokenEnum

template<typename T >

typedef T Sawyer::Lexer::Token< T >::TokenEnum

Definition at line 30 of file Lexer.h.

Constructor & Destructor Documentation

Token() [1/2]

template<typename T >

Sawyer::Lexer::Token< T >::Token ( )

inline

Construct an EOF token.

Definition at line 39 of file Lexer.h.

Token() [2/2]

template<typename T >

Sawyer::Lexer::Token< T >::Token ( TokenEnum  type, size_t  begin, size_t  end  )

inline

Construct a token.

The token has the specified type and its lexeme are the characters at the specified position in the input stream. The begin points to the first character of the lexeme and end points to one past the last character. A token's lexeme is allowed to be the empty string by specifying the same value for begin and end, however, end must never be less than begin.

Definition at line 47 of file Lexer.h.

Member Function Documentation

isEof()

template<typename T >

bool Sawyer::Lexer::Token< T >::isEof ( ) const

inline

Whether this is an EOF token.

An EOF token is a special token that has no type and an empty lexeme. EOF tokens are constructed by the default constructor.

Definition at line 56 of file Lexer.h.

type()

template<typename T >

TokenEnum Sawyer::Lexer::Token< T >::type ( ) const

inline

Returns the token.

Since EOF tokens have no type, this must not be called for an EOF token.

Definition at line 65 of file Lexer.h.

begin()

template<typename T >

size_t Sawyer::Lexer::Token< T >::begin ( ) const

inline

Token lexeme starting position.

This is the starting offset in the input for the first character of this token's lexeme.

Definition at line 72 of file Lexer.h.

end()

template<typename T >

size_t Sawyer::Lexer::Token< T >::end ( ) const

inline

Token lexeme ending position.

This is the offset in the input for one position past the last character of this token's lexeme. It is guaranteed to be greater than or equal to the begin position. EOF tokens will always have a begin equal to the end, but other empty non-EOF tokens are also possible.

Definition at line 81 of file Lexer.h.

---

The documentation for this class was generated from the following file:

• Lexer.h