Parse::Marpa::Lex -- Utility Methods for Lexing
These routines are used internally by MDL to implement lexing of regexes and of q- and qq-quoted strings.
They are documented here to make them available for general use within Marpa.
my ( $regex, $token_length ) =
Parse::Marpa::Lex::lex_regex( \$input_string, $lexeme_start );
Takes two required arguments. $string must be a reference to a string that might contain a regex. The regex will be expected to start at the position pointed to by pos $$string.
$lexeme_start must be the start earleme of the regex for lexing purposes. In many cases (such as the removal of leading whitespace), it's useful to discard prefixes. If a prefix was removed prior to the call to lex_regex, $lexeme_start should be the location where the prefix started. If no prefix was removed, $lexeme_start will be the same as pos ${$string}.
How lex_regex delimits a regex is described in the MDL document. lex_regex returns the null array if no regex was found. If a regex was found, lex_regex returns an array of two elements. The first element is a string containing the regex, its delimiters, any postfix modifiers it had, and its qr- "operator" if there was one. The second is the regex's length for lexing purposes, which will include the length of any discarded prefix.
my ( $string, $token_length ) =
Parse::Marpa::Lex::lex_q_quote( \$input_string, $lexeme_start );
Takes two required arguents, a string reference and a start earleme. The string reference must be to a string that might contain a q- or qq-quoted string. The q- or qq-quoted string will be expected to start at the position pointed to by pos ${$string}.
$lexeme_start must contain the start earleme of the quoted string for lexing purposes. In many cases (such as the removal of leading whitespace), it's useful to discard prefixes. If a prefix was removed prior to the call to lex_regex, $lexeme_start should be the location where the prefix started. If no prefix was removed, $lexeme_start should be the same as pos $$string.
How lex_q_quote delimits a q- or qq-quoted string is described in the MDL document. lex_q_quote returns the null array if no string was found. If a string was found, lex_q_quote returns an array of two elements. The first element is a string containing the q- or qq-quoted string, including the q- or qq- "operator" and the delimiters. The second is the quoted string's length for lexing purposes, which will include the length of any discarded prefix.
See the support section in the main module.
Jeffrey Kegler
Copyright 2007 - 2008 Jeffrey Kegler
This program is free software; you can redistribute it and/or modify it under the same terms as Perl 5.10.0.
