lestes::lang::cplus::lex::token_stream Class Reference

Stream of tokens. More...

#include <token_stream.hh>

Inheritance diagram for lestes::lang::cplus::lex::token_stream:

Inheritance graph
[legend]
List of all members.

Public Member Functions

ptr< pp_tokenread (void)
 Reads first token.
ptr< pp_tokenpeek_front (void)
 Returns front token.
ptr< pp_tokenread_front (void)
 Reads front token, squeezing whitespace.
ptr< pp_tokenread_front_skip_ws (void)
 Reads front token, skipping front whitespace, but not newline.
bool skip_front_ws (void)
 Skips front whitespace.
ptr< token_sequenceread_line (void)
 Reads next line in input.
ptr< token_sequenceexpand_line (const ptr< macro_storage > &macros)
 Expands next line in input.
void debug_print (::std::ostream &o)
 Prints debug dump.

Static Public Member Functions

static ptr< token_streamcreate (const ptr< pp_filter > &a_input)
 Returns stream with input.

Protected Member Functions

 token_stream (const ptr< pp_filter > &a_input)
 Creates stream.
void prepend (const ptr< token_sequence > &ts)
 Prepends a sequence.
virtual void gc_mark (void)
 Marks the object.

Private Types

typedef ::lestes::std::list<
srp< pp_token > > 
sequence_type
 The type of sequence container.

Private Member Functions

bool load_input (void)
 Loads next line.

Private Attributes

srp< pp_tokenfile_end
 The eof token.
srp< sequence_typesequence
 The sequence container.

Detailed Description

Stream of tokens.

Represents input stream of tokens, with access at front. The stream is connected to pp_filter to read tokens from. Never reads beyond the next newline, if not asked for.


Member Typedef Documentation

typedef ::lestes::std::list< srp<pp_token> > lestes::lang::cplus::lex::token_stream::sequence_type [private]

The type of sequence container.


Constructor & Destructor Documentation

lestes::lang::cplus::lex::token_stream::token_stream ( const ptr< pp_filter > &  a_input  )  [protected]

Creates stream.

Constructs new stream.

Precondition:
a_input != NULL
Parameters:
a_input The input to read from.


Member Function Documentation

ptr< pp_token > lestes::lang::cplus::lex::token_stream::read ( void   )  [virtual]

Reads first token.

Reads first token, does no transformation.

Returns:
The first token in the sequence, or TOK_TERMINATOR.

Implements lestes::lang::cplus::lex::pp_filter.

ptr< pp_token > lestes::lang::cplus::lex::token_stream::peek_front ( void   )  [virtual]

Returns front token.

Returns front token, which is kept in the sequence.

Returns:
The first token in the sequence, or TOK_TERMINATOR.

Implements lestes::lang::cplus::lex::token_input.

ptr< pp_token > lestes::lang::cplus::lex::token_stream::read_front ( void   )  [virtual]

Reads front token, squeezing whitespace.

Reads front token, squeezing multiple blanks into one.

Returns:
The first token in the sequence.

Implements lestes::lang::cplus::lex::token_input.

ptr< pp_token > lestes::lang::cplus::lex::token_stream::read_front_skip_ws ( void   )  [virtual]

Reads front token, skipping front whitespace, but not newline.

Reads front token, skipping front whitespace, but not newline.

Returns:
The first non-whitespace token in the sequence.

Implements lestes::lang::cplus::lex::token_input.

bool lestes::lang::cplus::lex::token_stream::skip_front_ws ( void   )  [virtual]

Skips front whitespace.

Skips front whitespace.

Returns:
true If there are some tokens left in the sequence.

Implements lestes::lang::cplus::lex::token_input.

ptr< token_sequence > lestes::lang::cplus::lex::token_stream::read_line ( void   ) 

Reads next line in input.

Reads line from the stream until TOK_LINE_END. At the end of stream returns the TOK_FILE_END.

Returns:
The nonexpanded tokens including the TOK_LINE_END, or sequence containing TOK_FILE_END.

ptr< token_sequence > lestes::lang::cplus::lex::token_stream::expand_line ( const ptr< macro_storage > &  macros  ) 

Expands next line in input.

Expands macros within sequence till newline. The first line, i.e. the front part ending with TOK_LINE_END, not counting TOK_LINE_END inside macro calls, is removed from the sequence.

Precondition:
macros != NULL
Parameters:
macros The macros for expansion.
Returns:
The expanded tokens including the TOK_LINE_END, or empty sequence.

void lestes::lang::cplus::lex::token_stream::debug_print ( ::std::ostream &  o  ) 

Prints debug dump.

ptr< token_stream > lestes::lang::cplus::lex::token_stream::create ( const ptr< pp_filter > &  a_input  )  [static]

Returns stream with input.

Returns a token stream connected to the a_input.

Precondition:
a_input != NULL
Parameters:
a_input The input to read from.
Returns:
New token stream.

void lestes::lang::cplus::lex::token_stream::prepend ( const ptr< token_sequence > &  inserted  )  [protected]

Prepends a sequence.

Inserts another token sequence before the beginning of the sequence.

Precondition:
inserted != NULL
Postcondition:
inserted->length() == 0
Parameters:
inserted The sequence to insert, the content is moved into new place.

void lestes::lang::cplus::lex::token_stream::gc_mark ( void   )  [protected, virtual]

Marks the object.

Marks the object.

Reimplemented from lestes::lang::cplus::lex::pp_filter.

bool lestes::lang::cplus::lex::token_stream::load_input ( void   )  [private]

Loads next line.

Loads next line delimited by pp_token::TOK_LINE_END from input into internal buffer. Expects pp_token::TOK_FILE_END to be returned after the last pp_token::TOK_LINE_END.

Returns:
true If before end of stream.


Member Data Documentation

srp<pp_token> lestes::lang::cplus::lex::token_stream::file_end [private]

The eof token.

srp<sequence_type> lestes::lang::cplus::lex::token_stream::sequence [private]

The sequence container.


The documentation for this class was generated from the following files:
Generated on Mon Feb 12 18:43:53 2007 for lestes by doxygen 1.5.1-20070107