Perl6::Tokener - It's a Perl 6 tokener. It tokenises Perl 6.
use Perl6::Tokener; my $t = new Perl6::Tokener(file=>"foo.pl", buffer => $code); while ($t->{buffer}) { my ($type, $token) = $t->toke(); ... }
I don't think there's really much I need to say about this. It isn't perfect, but I'm working on it. The synopsis pretty much gives you all you need to know to drive the thing, and, bluntly, if you're futzing with tokenising Perl 6, you're already beyond the need for most kinds of documentation. So have fun.
Oh, one thing - when you're parsing, you probably want to discard the type of everything called operator or term and just use the token value. Oh, and white space will return undef for token and type, so don't try using this in a while loop.
operator
term
undef
while
{ doesn't do what it ought. This is going to suck.
{
Some of the important keyword subs aren't implemented.
Simon Cozens, simon@cpan.org
simon@cpan.org
1 POD Error
The following errors were encountered while parsing the POD:
You forgot a '=back' before '=head1'
To install Perl6::Tokener, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Perl6::Tokener
CPAN shell
perl -MCPAN -e shell install Perl6::Tokener
For more information on module installation, please visit the detailed CPAN module installation guide.