package Pugs;







1;





### There is too much in this file 

### separate into:

- Introduction

- Documentation Index

- Compiler internals:

    Compiler

    Compiler::Perl6

    Compiler::Rule

    Compiler::YAML

    Compiler::Perl5

    Compiler::Parrot

- AST, Grammar, Optimizer internals:

    AST::Perl6::API

    Grammar::Perl6

    Optimizer::AST   -- optimizes an AST

    Grammar::MOP     -- implements OO using macros

    

=head1 Definitions



### change to kwid? - easier to indent (but not supported by p5 by default)

### move definitions to end-of-file?



* boxed / unboxed



* AST Abstract syntax tree

  - the data structure that the Parser generates

  (link to wikipedia?)



* DSL Domain-specific-language



...



=head1 The User Story



Say we have a Perl 6 snippet here:



    my $x = 'World';

    say "Hello, $x!";



Pugs::Parser gets it, turns it into a AST node (the API is

specified in Perl6::API::AST - so maybe the implementation

lives there too?)



Pugs::Compiler (or Analyzer?) gets it, emit a Module object

that defines ::Main.  A Module object mixes in both Package

and Code -- because it is both a namespace of symbols and has

a main body closure. 



the Package part of the Module is simply a hash from strings

(symbol names) to objects (containers)



the Code part consists of a Pad, a Sig and a Body.



the Pad contains My/State/Temp markers for container

allocation, as well as the compile-time-bound values for them.

It includes the parameters as well as anything declared within

that scope.



the Sig contains a simply structure of the signature; it does

not allocate containers.



the Body is just a structure with three node forms: Var

(variable lookup), App (application) and Lit (literal).



App is divided to Multiple, Single and Primitive dispatch.

Lit is divided to various intrinsic object types, including

intrinsic Code (which then has Pad/Sig/Body too)

Var is either a static lookup (lex only) or a dynamic lookup. 



This object-serialization format is shared across all runtimes.



=head1 Directory Layout



lib/Pugs/Compiler.pm  # functional things

lib/Perl6/API/Code.pm # userland objects -- those are written in p6 perhaps?



I propose: no lib6/ and lib/ distinctions -- .pm files should be both

p5 and p6 capable; p5 ones begin with



    package Foo;



and p6 ones begin with:



    use Perl-6.0;

    class Foo;



and we can stepwise replace p5 ones with p6 ones _without_ changings its name.



of course that has the problem of supplying Perl.pm itself - but I think to

do it Right As Specified is more important.



### - see "Modifying the compiler" below 



Pugs::Compiler::{name} ? for simultaneous versions



I'm not sure... I think Pugs::Compiler could work as the

basic interface, and if you'd like to offer more, subclass

it and call it Pugs::Compiler::Foo.



use Pugs::Compiler-fglock;

# this would load Pugs::Compiler::fglock as well as 

# all ::fglock things -- that is, Pugs::Compiler::SomeComponent::fglock

# as well -- and "use Pugs-fglock" will load all ::fglock things

# for Pugs::Compiler, Pugs::Emitter, etc?





I think subclassing is overrated, yeah -- mixins are much better

so lib/Pugs/Compiler.pm

but augmentations lives in somewhere else?

libX/fglock/Pugs/Compiler.pm?

lib/Pugs/Compiler.fglock.pm?



use Pugs-fglock; # everything is fglock

use Pugs qw( Parser fglock );



but where does the .pm live for real? I like per-user dirs



lib/fglock/Compiler.pm

lib/Pugs/Compiler.pm



### how about lib/Pugs-fglock/Compiler.pm

###   just like use Pugs-fglock above



=head1 Modifying the compiler



Let's say you have a new Grammar and Emitter for your own language.



  # assuming:

  # - your target system runs Perl5

  # - you have loaded the Grammar file to $new_parser

  # - and the emitter to $new_emitter

  

  # build a "normal" compiler that has: 

  # - a Perl6 Grammar with the Meta Object Protocol (implemented as macros), 

  #   Primitives,

  # - an optimizer chain,

  # - the default Perl6 libs,

  # - an emitter that generates perl5

  # load it as 'the' handler for 'eval( ... , :lang<perl6> )

  my $compiler = Pugs::Compiler->new( 

      lang      => 'perl6',

      parser    => ['Perl6', 'MOP6', 'PrimP6']

      optimizer => ['Op1' 'Op2'],

      lib       => ['/lib' ],

      emitter   => 'Perl5' );



  # same thing, but reads lib/Pugs/Compiler/Language/Perl6.yaml

  # this is the command used to implement 'eval( ... , :lang<name> )'

  my $compiler = Pugs::Compiler->new( 

      lang => 'Perl6' );



my $new_parser_p5 = $compiler->( $new_parser );

  

  # build a new compiler, that uses:

  # - the new Grammar

  # - no optimizer

  # - the new emitter

  my $compiler_OO_AST = Pugs::Compiler->load( 

      parser  => ['NewLang'],

      optimizer => [],

      emitter => \$new_emitter_p5 );

      

=head2 Example: a YAML compiler

      

* load the YAML compiler



  *compile::YAML = Pugs::Compiler->new( 

      lang      => 'YAML',

      parser    => ['YAML']

      optimizer => [],

      lib       => ['/lib' ],

      emitter   => 'YAML' );



  # use the configuration file (Pugs::Compiler::YAML.pm)

  *compile::YAML = Pugs::Compiler->new( lang => 'YAML' );



* use the compiler



  my $data = compile::eval( $source, lang => 'YAML' );



* Files



  /lib/Pugs/Compiler/YAML.pm  # configuration (like CPAN/Config.pm)

  /lib/Pugs/Grammar/YAML.pm   # main Grammar file

  /lib/Pugs/Grammar/YAML/*    # Grammar related files

  /lib/Pugs/Emitter/YAML.pm   # main Emitter file

  /lib/Pugs/Emitter/YAML/*    # Emitter related files



=head1 Compiler processes



### separate "processes" from actual module documentation?



=head2 Grammar engine



* Pugs::Grammar::Rule and Pugs::Grammar::Match



Rule constructors return Rule objects, which are "matcher" things.

Rules operate on strings (normally) and return Match objects.



### examples will be in p5 or p6?



  # invoking the rule compiler

### Rule::compile or Rule->compile? (assuming example in p5)

  my $matcher = Pugs::Grammar::Rule::compile ( ".*", options... );

  my $match = $matcher( "hello" );



### "internals" thing - should be documented only in Pugs::Grammar::Rule 

  # low level operators

  # Domain-specific-language-like syntax, contained in a package

### rule must ever use Rule::rule form - never Rule->rule ??? (assuming example in p5)

  my $matcher;   # must be declared outside of the block

  { package Pugs::Grammar::Rule;

      $matcher = rule ( alternation ( literal ( 'a' ), .... );

  }



Match objects (Pugs::Grammar::Match) contain:



### Match actually contain getters for $() and such

### getters need to know what you want (str, bool, hash, array, match)



- compile()



### obvious - not document ?

...



- rule()



### internals - move to module documentation



Rules, just like other Perl 6 values, exists in 'boxed' 

and 'unboxed' versions. 



The rule() constructor provides an interface between the 

low-level functional operators, and the higher level OO/Overload 

runtime. 



- internal rule operators



Internal rule operators share a common API, which permits the intercommunication 

of data about: backtracking, aborting, capturing.



There are 2 kinds of ruleop operators:



- higher order operators: alternation, concatenation.

- "plain" match operators: like 'literal'.



Higher order operators take a list of match operators, and return a new 

match operator.



A ruleop() function gets as argument a list:



### we don't need a "high" level interface (named args) 

###    because the actual rule object will be full OO



:(Str $match_against, :$next_state, Bool :$capture?

  --> Pugs::Grammar::Match)



  $match_against - a string to match

  $next_state    - an optional "continuation"

  $capture       - capture then entire match. For example:

  

  # returns a Match object that has the 'capture' field set to 'abc'

  match_word( 'abc123', :capture );



  # returns a Match object that has no capture (but the 'bool' field is still true)

  match_word( 'abc123' );



A ruleop function returns:



    # spec says it returns normal data, but this is a low-level op...

    undef - if match failed    



    state - a "next-state" or undef

    bool - an "assertion" (true/false)

    match - the "match" tree or undef

    tail - the string tail or undef

    capture - the tree of captured things

    abort - the match was stopped by a { return } or a fail(),

           and it should not backtrack or whatever

                            

=for notes about fail()

shouldn't matchfail still be returning something?

maybe just overload bool to false ;)

(I mean, seriously, because that's what p6 does)

and also you can overload @{} %{} &{} + ~



I think it should be magical at "unboxed" level yes.

I mean, if I use Pugs::Grammar in p5 code

I'd _expect_ the match object returned to carry the same magic.



overload in p5 is just static %OVERLOAD entry

the only overhead is a SVMG (i.e. bless())

you can return the same failure object always

then there's absolutely no overhead

I'm not sure about this - failed matche's .end is undefined

=cut



### current implementation is always anchored at ^

### because it was specialized for subrules

### - needs to be generalized, and the anchor should only

###   work when needed (or if there is a switch telling so)