NAME

Parse::Token::Lite - Simply parse String into tokens with rules which are similar to Lex.

VERSION

version 0.200

SYNOPSIS

        use Parse::Token::Lite;

        my %rules = (
                MAIN=>[
                        { name=>'NUM', re=> qr/\d[\d,\.]*/ },
                        { name=>'STR', re=> qr/\w+/ },
                        { name=>'SPC', re=> qr/\s+/ },
                        { name=>'ERR', re=> qr/.*/ },
                ],
        );

        my $parser = Parse::Token::Lite->new(rulemap=>\%rules);
        $parser->from("This costs 1,000won.");
        while( ! $parser->eof ){
                my ($token,@extra) = $parser->nextToken;
                print $token->rule->name."-->".$token->data."<--\n";
        }

Results are

        STR -->This<--
        SPC --> <--
        STR -->costs<--
        SPC --> <--
        NUM -->1,000<--
        STR -->won<--
        ERR -->.<--

ATTRIBUTES

rulemap

rulemap contains hash refrence of rule objects grouped by STATE. rulemap should have 'MAIN' item.

        my %rule = (
                MAIN => [
                        Parse::Token::Lite::Rule->new(name=>'any', re=>qr/./),
                ],
        );
        $parser->rulemap(\%rule);

In constructor, it can be replaced with hash reference descripting attributes of Parse::Token::Lite::Rule class, intead of Rule Object.

        my %rule = (
                MAIN => [
                        {name=>'any', re=>qr/./}, # ditto
                ],
        );
        my $parser = Parse::Token::Lite->new( rulemap=>\%rule );

data

'data' is set by from() method. 'data' contains a rest of text which is not processed by nextToken(). Please remember, 'data' is changing.

If a length of 'data' is 0, eof() returns 1.

state_stack

At first time, it contains ['MAIN']. It is reset by from().

METHODS

from($data_string)

Setting data to parse.

This causes resetting state_stack.

parse()

parse($data)

On Scalar context : Returns 1 On Array context : Returns array of [Parse::Token::Lite::Token,@return_values_of_callback].

Parse all tokens on Event driven. Just call nextToken() during that eof() is not 1.

Defined $data causes calling from($data).

You should set a callback function at 'func' attribute in 'rulemap' to do something with tokens.

currentRules()

Returns an array reference of rules of current state.

See Parse::Token::Lite::Rule.

nextToken()

On Scalar context : Returns Parse::Token::Lite::Token object. On Array context : Returns (Parse::Token::Lite::Token,@return_values_of_callback).

        my ($token, @ret) = $parser->nextToken;
        print $token->rule->name . '->' . $token->data . "\n";

See Parse::Token::Lite::Token and Parse::Token::Lite::Rule.

eof()

Returns 1 when no more text is.

start($state)

end()

end($state)

Push/Pop the state on state_stack to implement AUTOMATA.

Also, this is called by a 'state' definition of Parse::Token::Lite::Rule.

You can set rules as Lexer like.

        my $rulemap = {
                MAIN => [
                        { name=>'QUOTE', re=>qr/'/, func=>
                                sub{ 
                                        my ($parser,$token) = @_;
                                        $parser->start('STATE_QUOTE'); # push
                                }
                        },
                        { name=>'ANY', re=>qr/.+/ },
                ],
                STATE_QUOTE => [
                        { name=>'QUOTE_PAIR', re=>qr/'/, func=>
                                sub{ 
                                        my ($parser,$token) = @_;
                                        $parser->end('STATE_QUOTE'); # pop
                                }
                        },
                        { name=>'QUOTED_TEXT', re=>qr/.+/ }
                ],
        };

You can also do it in simple way.

        my $rulemap = {
                MAIN => [
                        { name=>'QUOTE', re=>qr/'/, state=>['+STATE_QUOTE'] }, # push
                        { name=>'ANY', re=>qr/.+/ },
                ],
                STATE_QUOTE => [
                        { name=>'QUOTE_PAIR', re=>qr/'/, state=>['-STATE_QUOTE] }, #pop
                        { name=>'QUOTED_TEXT', re=>qr/.+/ }
                ],
        };

state()

Returns current state by peeking top of 'state_stack'.

SEE ALSO

See Parse::Token::Lite::Token and Parse::Token::Lite::Rule.

And see 'samples' directory in source.

AUTHOR

khs <sng2nara@gmail.com>

COPYRIGHT AND LICENSE

This software is copyright (c) 2013 by khs.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

cpanm Parse::Token::Lite

perl -MCPAN -e shell
install Parse::Token::Lite