Paulo Custodio > Parse-FSM-1.07 > Parse::FSM::Lexer

Download:
Parse-FSM-1.07.tar.gz

Dependencies

Annotate this POD

CPAN RT

Open  0
View/Report Bugs
Module Version: 1.07   Source   Latest Release: Parse-FSM-1.10

NAME ^

Parse::FSM::Lexer - Companion Lexer for the Parse::FSM parser

SYNOPSIS ^

  use Parse::FSM::Lexer;
  $lex = Parse::FSM::Lexer->new;
  $lex = Parse::FSM::Lexer->new(@files);
  
  $lex->add_path(@dirs); @dirs = $lex->path;
  $full_path = $lex->path_search($file);

  $lex->from_file($filename);
  $lex->from_list(@input); 
  $lex->from_list(sub {});
  
  $lex->get_token;
  
  $lex->error(@message); 
  $lex->file; 
  $lex->line_nr;
  
  # in a nearby piece of code
  use MyParser; # isa Parse::FSM::Driver;
  my $parser = MyParser->new;
  $parser->input(sub {$lex->get_token});
  eval {$parser->parse}; $@ and $lex->error($@);

DESCRIPTION ^

This module implements a generic tokenizer that can be used by Parse::FSM parsers, and can also be used stand alone independently of the parser.

It supports recursive file includes and takes track of current file name and line number. It keeps the path of search directories to search for input files.

The get_token method can be called by the input method of the parser to retrieves the next input token to parse.

The module can be used directly if the supplied tokenizer is enough for the application, but usually a derived class has to be written implementing a custom version of the tokenizer method.

METHODS - SETUP ^

new

Creates a new object. If an argument list is given, calls from_file for each of the file starting from the last, so that the files are read in the given order.

METHODS - SEARCH PATH FOR FILES ^

path

Returns the list of directories to search in sequence for source files.

add_path

Adds the given directories to the path searched for include files.

path_search

Searches for the given file name in the path created by add_path, returns the first full path name where the file can be found.

Returns the given input file name unchanged if:

METHODS - INPUT STREAM ^

from_file

Saves the current input context, searches for the given input file name in the path, opens the file and sets-up the object to read each line in sequence. At the end of the file input resumes to the place where it was when from_file was called.

Dies if the input file cannot be read, or if a file is included recursively, to avoid an infinite include loop.

from_list

Saves the current input context and sets-up the object to read each element of the passed input list. Each element either a text string or a code reference of an iterator that returns text strings. The iterator returns undef at the end of input.

METHODS - INPUT ^

get_token

Retrieves the next token from the input as an array reference containing token type and token value.

Returns undef on end of input.

tokenizer

Method responsible to match the next token from the given input string.

This method can be overriden by a child class in order to implement a different set ot tokens to be retrieved from the input.

It is implemented with features from the Perl 5.010 regex engine:

The default tokenizer recognizes and returns the following token types:

[STR => $value]

Perl-like single or double quoted string, $value contains the string without the quotes and with any backslash escapes resolved.

The string cannot span multiple input lines.

[NUM => $value]

Perl-like integer in decimal, hexadecimal, octal or binary notation, $value contains decimal value of the integer.

[NAME => $name]

Perl-like identifier name, i.e. word starting with a letter or underscore and followed by letters, underscores or digits.

[$token => $token]

All other characters except white space are returned in the form [$token=>$token], where $token is a single character or one of the following composed tokens: << >> == != >= <=

white space

All white space is ignored, i.e. the tokenizer returns undef.

[INCLUDE => $file]

Returned when a #include statement is recognized, causes the lexer to recursively include the file at the current input stream location.

[INPUT_POS => $file, $line_nr, $line_inc]

Returned when a #line statement is recognized, causes the lexer to set the current input location to the given $file, $line_nr and $line_inc.

[ERROR => $message]

Causes the lexer to call error with the given error message, can be used when the input cannot be tokenized.

METHODS - INPUT LOCATION AND ERRORS ^

file

Returns the current input file, undef if reading from a list.

line_nr

Returns the current input line number, starting at 1.

line_inc

Increment of line number on each new-line found, usually 1.

error

Dies with the given error message, indicating the place in the input source file where the error occured.

warning

Warns with the given error message, indicating the place in the input source file where the warning occured.

AUTHOR, BUGS, FEEDBACK, LICENSE, COPYRIGHT ^

See Parse::FSM

syntax highlighting: