package Perl::Lexer;
use 5.018001;
use strict;
use warnings;
use B;
our $VERSION = "0.02";
use parent qw(Exporter);
our @EXPORT = qw(
TOKENTYPE_NONE
TOKENTYPE_IVAL
TOKENTYPE_OPNUM
TOKENTYPE_PVAL
TOKENTYPE_OPVAL
);
use Perl::Lexer::Token;
use XSLoader;
XSLoader::load(__PACKAGE__, $VERSION);
sub new {
my $class = shift;
bless {}, $class;
}
sub scan_string {
my ($self, $str) = @_;
open my $fh, '<', \$str;
$self->scan_fh($fh);
}
4649;
__END__
=for stopwords hackish yylval tokenize
=encoding utf-8
=head1 NAME
Perl::Lexer - Use Perl5 lexer as a library.
=head1 SYNOPSIS
use v5.18.0;
use Perl::Lexer;
my $lexer = Perl::Lexer->new();
my @tokens = @{$lexer->scan_string('1+2')};
for (@tokens) {
say $_->inspect;
}
Output is:
<Token: THING opval 1>
<Token: ADDOP opnum 63>
<Token: THING opval 2>
=head1 DESCRIPTION
B<THIS LIBRARY IS WRITTEN FOR RESEARCHING PERL5 LEXER API. THIS MODULE USES PERL5 INTERNAL API. DO NOT USE THIS.>
Perl::Lexer is a really hackish library for using Perl5 lexer as a library.
=head1 MOTIVATION
The programming language provides lexer library for itself is pretty nice.
I want to tokenize perl5 code by perl5.
Of course, this module uses Perl5's private APIs. I hope these APIs turn into public.
If we can use lexer, we can write a source code analysis related things like Test::MinimumVersion, and other things.
=head1 WHAT API IS NEEDED FOR WRITING MODULES LIKE THIS.
=over 4
=item Open the token informations for XS hackers.
Now, token name, type, and arguments informations are hidden at toke.c and perly.h.
I need to define `PERL_CORE` for use it... It's too bad.
And, I take token_type and debug_tokens from toke.c.
=back
=head1 METHODS
=over 4
=item my $lexer = Perl::Lexer->new();
Create new Perl::Lexer object.
=item $lexer->scan_string($code: Str) : ArrayRef[Str]
Tokenize perl5 code. This method returns arrayref of Perl::Lexer::Token.
=back
=head1 LICENSE
Copyright (C) Tokuhiro Matsuno.
This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.
=head1 AUTHOR
Tokuhiro Matsuno E<lt>tokuhirom@gmail.comE<gt>
=cut