MarpaX::Repa::Lexer - simplify lexing for Marpa parser
Most details are in MarpaX::Repa.
Returns a new lexer instance. Takes named arguments.
my $lexer = MyLexer->new( tokens => { word => qr{\b\w+\b}, }, store => 'array', debug => 1, );
Possible arguments:
Hash with names of terminals as keys and one of the following as value:
Just a string to match.
'a token' => "matches this long string",
A qr{} compiled regexp.
qr{}
'a token' => qr{"[^"]+"},
Note that regexp MUST match at least one character. At this moment look behind to look at chars before the current position is not supported.
With hash you can define token specific options. At this moment 'store' option only (see below). Use match key to set what to match (string or regular expression):
match
'a token' => { match => "a string", store => 'hash', },
Per token options are:
What to store (pass as value to Marpa's recognizer). The following variants are supported:
{ token => 'a token', value => 'a value' }
[ 'a token', 'a value' ]
'a value'
undef is stored so later Repa's actions will skip it.
A function will be called with token name and reference to its value. Should return a reference or undef that will be passed to recognizer.
A callback that can check whether token is really match or not.
If true then parse should complete in one go and consume whole input.
If true then lexer prints debug log to STDERR.
Minimal size of the buffer (4*1024 by default).
Setups instance and returns $self. No need to call, it's called from the constructor.
$self
Takes a recognizer and a file handle. Parses input. Dies on critical errors, but not when parser lost its way. Returns recognizer that was passed.
Returns reference to the current buffer.
Called when "buffer" needs a re-fill with a file handle as argument. Returns true if there is still data to come from the handle.
Returns first 20 chars of the buffer with everything besides ASCII encoded with \x{####}. Use argument to control size, zero to mean whole buffer.
\x{####}
To install MarpaX::Repa, copy and paste the appropriate command in to your terminal.
cpanm
cpanm MarpaX::Repa
CPAN shell
perl -MCPAN -e shell install MarpaX::Repa
For more information on module installation, please visit the detailed CPAN module installation guide.