NAME

Devel::Tokenizer::C - Generate C source for fast keyword tokenizer

SYNOPSIS

  use Devel::Tokenizer::C;
  
  $t = new Devel::Tokenizer::C TokenFunc => sub { "return \U$_[0];\n" };
  
  $t->add_tokens( qw( bar baz ) )->add_tokens( ['for'] );
  $t->add_tokens( [qw( foo )], 'defined DIRECTIVE' );
  
  print $t->generate;

DESCRIPTION

The Devel::Tokenizer::C module provides a small class for creating the essential ANSI C source code for a fast keyword tokenizer.

The generated code is optimized for speed. On the ANSI-C keyword set, it's 2-3 times faster than equivalent code generated with the gprof utility.

The above example would print the following C source code:

  switch( tokstr[0] )
  {
    case 'b':
      switch( tokstr[1] )
      {
        case 'a':
          switch( tokstr[2] )
          {
            case 'r':
              if( tokstr[3] == '\0' )
              {                                     /* bar        */
                return BAR;
              }
  
              goto unknown;
  
            case 'z':
              if( tokstr[3] == '\0' )
              {                                     /* baz        */
                return BAZ;
              }
  
              goto unknown;
  
            default:
              goto unknown;
          }
  
        default:
          goto unknown;
      }
  
    case 'f':
      switch( tokstr[1] )
      {
        case 'o':
          switch( tokstr[2] )
          {
  #if defined DIRECTIVE
            case 'o':
              if( tokstr[3] == '\0' )
              {                                     /* foo        */
                return FOO;
              }
  
              goto unknown;
  #endif /* defined DIRECTIVE */
  
            case 'r':
              if( tokstr[3] == '\0' )
              {                                     /* for        */
                return FOR;
              }
  
              goto unknown;
  
            default:
              goto unknown;
          }
  
        default:
          goto unknown;
      }
  
    default:
      goto unknown;
  }

So the generated code only includes the main switch statement for the tokenizer. You can configure most of the generated code to fit for your application.

CONFIGURATION

TokenFunc => SUBROUTINE

A reference to the subroutine that returns the code for each token match. The only parameter to the subroutine is the token string.

This is the default subroutine:

  TokenFunc => sub { "return $_[0];\n" }

TokenString => STRING

Identifier of the C character array that contains the token string. The default is tokstr.

UnknownLabel => STRING

Label that should be jumped to via goto if there's no keyword matching the token. The default is unknown.

TokenEnd => STRING

Character that defines the end of each token. The default is the null character '\0'.

CaseSensitive => 0 | 1

Boolean defining whether the generated tokenizer should be case sensitive or not. This will only affect the letters A-Z. The default is 1, so the generated tokenizer is case sensitive.

ADDING TOKENS

You can add tokens using the add_tokens method.

The method either takes a list of token strings or a reference to an array of token strings which can optionally be followed by a preprocessor directive string.

Calls to add_tokens can be chained together, as the method returns a reference to its object.

GENERATING THE CODE

The generate method will return a string with the tokenizer switch statement. If no tokens were added, it will return an empty string.

AUTHOR

Marcus Holland-Moritz <mhx@cpan.org>

BUGS

I hope none, since the code is pretty short. Perhaps lack of functionality ;-)

COPYRIGHT

Copyright (c) 2003, Marcus Holland-Moritz. All rights reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

cpanm Devel::Tokenizer::C

perl -MCPAN -e shell
install Devel::Tokenizer::C