pX/Common/iterator_engine/TASKS - fglock

Bootstrapping:
ok - core rule engine - backtracking, function composition - in Perl 5
ok - Perl 6 rules in Perl 5 - reuse to write itself
ok - Perl 6 grammar in Perl 5 - reuse to write itself
ok - Perl 6 to Perl 5 translator (parse p6, emit p5) - reuse to write 
     itself
   - Perl 6 compiler written in p6
      - emitted to p5
   - Integrate with Pugs features 
      - see discussions in #perl6, Notes, Deal.txt

Next possible milestone:
ok - rewrite iterator_engine.pl in Perl 6, 
   - compile to Perl 5,
   - use it instead of the original iterator_engine.pl

In order to implement new syntax:
  1 - optimize macro expansion 
  2 - or: implement statements in perl5 (as new nodes)
  3 - keep using macros as-is, and optimize later

  - option 1 is clean, but hard to do.
  - option 2 is unclean, easier to do, but the final result in speed 
    is about the same.
  - option 3 is good, but may get unusable because of slowness

Current tasks:
- make an automatic backup of the previous compiler before building a
  new prelude
- migrate all remaining rules from p5 to p6 (only a few last)
- implement 'macro' and '<%hash>', in order to define 'if' as in S04-Statement parsing
- organize grammar categories: statement_control, prefix, infix
- implement operator precedence
- migrate primitives from http://svn.openfoundry.org/pugs/src/Pugs/Prim.hs
- find out more about grammar bootstrapping and the parse/compile/run cycle and BEGIN/CHECK/INIT
- see TODO and XXX notes everywhere
- split Prelude into base-syntax and primitives
- fix Perl 6 syntax errors in Prelude - usage of $<>, ...

More tasks:
- implement syntax options:
  - default parser for macros
  - short operator names infix:<+> === +
- use putter's Match.pm
- bind special variables $0, ...
- see also the 'not implemented' rule list below
- optimize infix:<+> to 'native +' when possible
- mixing boxed/unboxed types (there are several tools available - which one to use?)
- Perl 6 OO - which tools to use? - talk to stevan
- un-source-filter rule return blocks
- un-source-filter macro variable binding
- coroutines

Other interesting things to do (low priority):
- reimplement iterator_engine.pl in 
  - Javascript - I like the 'run in the browser' idea very much
  - Perl 6 - Perl 6 version could be started with a source filter, since the syntax used is very simple
  - Parrot - not so useful - Parrot already has PGE
- test the compiler in Perl 5.5x
- try out Smart::Comments
- move project to a new subdir and give it a name?
- Extract 'Changes' and 'TODO' from README (extract old Changes from svn)
- make rule/grammar a pluggable module (pack for CPAN?)
  how about "Perl6::Grammar", and reuse Parse::RecDescent API?
- make grammars inherit from Grammar; make grammars inheritable
- write an emitter that generates perl5 regexes (or dies)
- implement "Rule|Str" with caching in <$var> rule
- pluggable emitters
- emit Pugs-compatible YAML 
- import Pugs-generated YAML precompiled code
- regex options :s :Perl5 ...

Possible difficult problems to fix (all untested):
- package-wide lexical variables (incremental compilation and execution doesn't work with lexicals)
- modifying the grammar from inside a block (incremental compilation is currently disabled inside blocks)
- clonable closures (needed for proper working lazy lists)

Cleanup namespaces:
  - find out what are the right names, such as:
  Grammar - the base grammar - defines <ws>, <digit>
  RuleGrammar - the pattern matching grammar (or ???)
  Perl6Grammar - the Perl6 grammar (or Perl6 ???)

---------------------------------

From p6compiler.pl -

# XXX - make grammars inherit from Grammar; make grammars inheritable
# XXX - write an emitter that generates perl5 regexes (or dies)
# XXX - add (API/documentation) to generate unnamed rules, unnamed 
grammars
# XXX - fix the extra commas in the generated code
# XXX - create error messages for compiling errors

=for later
        rule sub_application {
            <@grammar1::terms> <ws>? <@grammar1::ops> <ws>? <@grammar1::terms>
        }
        push @terms, \&sub_application;
    
        # XXX - this doesn't work
        #       say sub { print 'anonymous sub'; } ;
        rule anon_sub {
            sub <block>
                { return { anon_sub => $<block> ,} }
        }
        push @terms, \&anon_sub;

        rule assignment {
            $lvalue := (<variable>) <ws>? \= <ws>? $rvalue := (<variable>) <ws>? \;
                { return { assignment => [ $<lvalue>, $<rvalue> ] ,} }
        }
        unshift @terms, \&assignment;
        rule eval_perl5 {
            eval <ws>? \( <ws>? \" <code> \" <ws>? \, <ws>? \:lang\<perl5\> <ws>? \) <ws>? \;
        }
    # sub print ... 
    # sub 'Y' - is assoc 'list'
    # sub infix:<+> { eval( '$_[0]+$_[1]', :lang<perl5> ) }

    #        print '1' + '1';
    #    $a = $b;
    # TODO - $a = $b; - see 'rule assignment' above
    # TODO - rule comment { \# .*? [<newline>|$$] }
=cut

--- from file p6prelude.pl

=for TODO
    reimplement print(), warn ... using 'sub'
    implement eval_perl6 and eval_block

    operand fixity (infix, prefix...)
    operand precedence (+, *, ln)

    class

    find out how to change syntax while in the parse-and-generate-ast 
phase
    (for example, when a new sub is created)

    macros
=cut