View on
MetaCPAN is shutting down
For details read Perl NOC. After June 25th this page will redirect to
Simon Cozens > Perl6-Tokener-0.01 > Perl6::Tokener



Annotate this POD

View/Report Bugs
Module Version: 0.01   Source  


Perl6::Tokener - It's a Perl 6 tokener. It tokenises Perl 6.


    use Perl6::Tokener;
    my $t = new Perl6::Tokener(file=>"", buffer => $code);
    while ($t->{buffer}) {
        my ($type, $token) = $t->toke();


I don't think there's really much I need to say about this. It isn't perfect, but I'm working on it. The synopsis pretty much gives you all you need to know to drive the thing, and, bluntly, if you're futzing with tokenising Perl 6, you're already beyond the need for most kinds of documentation. So have fun.

Oh, one thing - when you're parsing, you probably want to discard the type of everything called operator or term and just use the token value. Oh, and white space will return undef for token and type, so don't try using this in a while loop.



Simon Cozens,

syntax highlighting: