Template::Benchmark::Engine - Base class for Template::Benchmark template engine plugins.
package Template::Benchmark::Engines::TemplateSandbox; use warnings; use strict; use base qw/Template::Benchmark::Engine/; our $VERSION = '0.99_02'; our %feature_syntaxes = ( literal_text => join( "\n", ( join( ' ', ( 'foo' ) x 12 ) ) x 5 ), scalar_variable => '<: expr scalar_variable :>', ); # rest of module...
Provides a base class for Template::Benchmark template engine plugins.
To write your own Template::Benchmark plugin you'll need to subclass this class (Template::Benchmark::Engine) and put your package in the Template::Benchmark::Engines:: namespace.
Template::Benchmark::Engines::
The naming convention within that namespace is to strip the :: from the name of the template engine and retain capitalization, thus Template::Sandbox becomes plugin Template::Benchmark::Engines::TemplateSandbox, HTML::Template becomes Template::Benchmark::Engines::HTMLTemplate.
The notable exception is that Template becomes Template::Benchmark::Engines::TemplateToolkit, because everyone calls it Template::Toolkit rather than Template.
Throughout the sections below are references to whether a template feature or cache type is supported or unsupported in the template engine.
Indicating that something is unsupported is fairly simple, you just return an undef value in the appropriate place, but what constitutes "unsupported"?
undef
It doesn't neccessarily mean that it's impossible to perform that task with the given template engine, but generally if it requires some significant chunk of DIY code or boilerplate or subclassing by the developer using the template engine, it should be considered to be unsupported by the template engine itself.
This of course is a subjective judgement, but a general rule of thumb is that if you can tell the template engine to do it, it's supported; and if the template engine allows you to do it, it's unsupported, even though it's possible.
Your plugin doesn't need to provide this method directly, it can be inherited from Template::Benchmark::Engine where it will access the %feature_syntaxes variable in your plugin's namespace, using $template_feature as a key.
Obviously %feature_syntaxes can't be a private variable for this to work, so declare it as a global or with our.
our
For example:
our %feature_syntaxes = ( literal_text => join( "\n", ( join( ' ', ( 'foo' ) x 12 ) ) x 5 ), scalar_variable => '<: expr scalar_variable :>', hash_variable_value => '<: expr hash_variable.hash_value_key :>', # ... );
Any feature that isn't supported by the template engine should have an undef value.
Please see the section "Feature Syntaxes" for a list of the different template features and what their requirements are.
After the template has been generated from the snippets for each of the enabled template features, it is passed to Plugin->preprocess_template() in case the plugin needs to do any final changes to the template.
Plugin->preprocess_template()
In the majority of cases the default preprocess_template() will be sufficient, but for some template engines which require unique labels for their loops, there may need to be a rewrite pass done here to ensure that the copies inserted by the template_repeats parameter have unique names.
preprocess_template()
template_repeats
preprocess_template() should not perform any template execution, it is simply a stage to ensure that a well-formed template is generated from the feature snippets.
This method was added in version 1.08.
This method must be defined, to return a hashref of benchmark function names to a useful description, for example:
sub benchmark_descriptions { return( { TS => "Template::Sandbox ($Template::Sandbox::VERSION) without caching", TS_CF => "Template::Sandbox ($Template::Sandbox::VERSION) with " . "Cache::CacheFactory ($Cache::CacheFactory::VERSION) caching", TS_CHI => "Template::Sandbox ($Template::Sandbox::VERSION) with " . "CHI ($CHI::VERSION) caching", TS_FMM => "Template::Sandbox ($Template::Sandbox::VERSION) with " . "Cache::FastMmap ($Cache::FastMmap::VERSION) caching", } ); }
It's generally very useful to include version numbers like the example, don't waste half an hour figuring out why your benchmarks are slow before you realise it's benchmarking the last version from CPAN rather than your fancy new development copy, all because you forgot to set your library path.
Uh, not that I'd make such a basic mistake.
The name of the different benchmarks shouldn't clash with anyone else's otherwise things will get confusing, and an underscore (_) should only be used to distinguish between different functions returned by the same plugin.
Best bet is to look at the plugins already written and choose something written along similar lines, but distinctive to your template engine, it needs to be short though because it will be being used as a column heading and row title for the benchmark results, and they're pretty wide already.
The convention I've used is to use initials of the package's namespace, T for Template::, H for HTML::, Te for Text::, then sufficient initials to disambiguate the rest of the package name.
So Template::Toolkit is TT, Template::Sandbox is TS, HTML::Template is HT, Text::Template is TeTe as opposed to Text::Tmpl being TeTmpl.
Additional initials might be added if the template engine can accept different template syntaxes, which is handled by several plugins:
Template::Alloy gets TATT (running in Template::Toolkit mode) and TAHT (running in HTML::Template mode), and if you want long-winded there's TeMMTeTe - Text::MicroMason running in Text::Template mode.
It should be obvious by now why there needs to be a nice clear description to accompany these names.
Within a plugin, if there are several different configuration options, caching choices, or other tweaks to be benchmarked, this is indicated by a suffix after an underscore (_).
So Template::Sandbox using Cache::CacheFactory for caching becomes TS_CF, and using CHI for caching is TS_CHI, this lets you easily see within the results that both TS_CF and TS_CHI are produced using the same template engine plugin.
These methods need to return a hashref of names to benchmark function references, if the cache type is unsupported it should return undef.
Each name needs to be listed in the hashref returned from Plugin->benchmark_descriptions().
Plugin->benchmark_descriptions()
For uncached_string, each benchmark function needs to accept the contents of the template as the first argument and then two hashrefs of template variables to set.
For uncached_disk, each benchmark function needs to accept the leaf filename of the template as the first argument and then two hashrefs of template variables to set.
The benchmark function should return the content of the processed template.
sub benchmark_functions_for_uncached_string { my ( $self ) = @_; return( { TS => sub { my $t = Template::Sandbox->new(); $t->set_template_string( $_[ 0 ] ); $t->add_vars( $_[ 1 ] ); $t->add_vars( $_[ 2 ] ); ${$t->run()}; }, } ); }
Please see the section "Cache Types" for a list of the different cache types and what restrictions apply to the benchmark functions in each.
Each of these methods need to return a hashref of names to benchmark function references like Plugin->benchmark_functions_for_uncached_string(), however they have slightly different arguments. If the cache type is unsupported it should return undef.
Plugin->benchmark_functions_for_uncached_string()
$template_dir provides you with the location of the temporary directory in which the template will be written.
$cache_dir provides you with the location of the temporary directory created for any cache you wish to create.
Note that the cache directory is unique to your plugin, however it is shared for every benchmark function returned by your plugin, if you have multiple benchmark functions you will need to set things up yourself within that cache directory to prevent them stomping on each-other's toes.
Each benchmark function needs to accept the leaf filename of the template as the first argument and then two hashrefs of template variables to set.
The leaf filename is the final bit after the directory if you're wondering, so if the template was /tmp/KJJFKav/TemplateSandbox/TemplateSandbox.txt then you'd get /tmp/KJJFKav/TemplateSandbox as $template_dir, and TemplateSandbox.txt passed as the first argument to your benchmark function.
/tmp/KJJFKav/TemplateSandbox/TemplateSandbox.txt
/tmp/KJJFKav/TemplateSandbox
$template_dir
TemplateSandbox.txt
sub benchmark_functions_for_disk_cache { my ( $self, $template_dir, $cache_dir ) = @_; my ( $cf, $chi ); $cf = Cache::CacheFactory->new( storage => { 'file' => { cache_root => $cache_dir, }, }, ); $chi = CHI->new( driver => 'File', root_dir => $cache_dir, ); return( { TS_CF => sub { my $t = Template::Sandbox->new( cache => $cf, template_root => $template_dir, template => $_[ 0 ], ignore_module_dependencies => 1, ); $t->add_vars( $_[ 1 ] ); $t->add_vars( $_[ 2 ] ); ${$t->run()}; }, TS_CHI => sub { my $t = Template::Sandbox->new( cache => $chi, template_root => $template_dir, template => $_[ 0 ], ignore_module_dependencies => 1, ); $t->add_vars( $_[ 1 ] ); $t->add_vars( $_[ 2 ] ); ${$t->run()}; }, } ); }
This informative method should return the type of syntax this template engine uses. Broadly speaking, most template engines fall into either the 'mini-language' or 'embedded-perl' camps, so return one of those two strings.
This informative method should return 1 if the template engine is written in pure perl, and 0 if the engine makes use of XS code, is a wrapper around a C library or in some other way mandates the use of non-perl dependencies.
1
0
The default method returns undef and will treat the engine as not being pure-perl. This may raise a warning or error in future versions.
If a plugin has several benchmark names, some pure-perl and some otherwise, this method should return a hashref of name to 0 or 1 for the respective answers.
For example, from Template::Benchmark::Engines::TemplateTolkit:
sub pure_perl { return( { TT => 1, TT_X => 0, TT_XCET => 0, } ); }
Comparing a template engine that's running with a memory cache to a completely uncached engine is like comparing apples with oranges, so each cache type is designed to simulate a different environment in which the template engine is running, this lets Template::Benchmark group results so that a fair comparison can be made between engines at performing a similar task.
With this in mind, each cache type has its own restrictions that should be adhered to when writing a plugin.
Common to each cache type is the requirement to accept two seperate hashrefs of template variables, and behave as if they might be different between invocations of the benchmark function. (Currently the contents of the variable hashrefs do not change, however that may change in future versions, and regardless, they should be treated as if they have changed each time.)
This cache type explicitly disallows caching of any kind, and must take the template as the supplied scalar value and process it "from scratch" each time.
This broadly simulates running in an uncached CGI environment if you're thinking of web applications, or the performance of a cache-miss in a cached environment.
This cache type explicitly disallows caching of any kind, and must take the template as the contents of the supplied filename, read it from disk freshly each time and process it "from scratch" each time.
This cache type requires that the template be read from disk, from the filename given, and may cache intermediate stages on the disk too.
No template data may be kept in-memory between invocations.
This broadly simulates running in a CGI environment with a disk cache to store compiled templates between requests.
This cache type requires that the template be read from disk, from the filename given, and may cache intermediate stages in shared memory.
No template data may be kept in non-shared memory between invocations.
It's quite normal for template engines not to provide shared memory support, so not many plugins provide this cache type.
This broadly simulates running in a mod_perl environment with the templates loaded into a shared memory cache before the webserver forks.
This cache type requires that the template be read from disk, from the filename given, and may cache intermediate stages in memory.
While template data may be stored in-memory, it must be accessed by instantiating a new copy of the template engine with the provided filename, and not simply by reusing an instance or compiled subroutine reference from a previous run.
ie: The benchmark function itself should store no stateful information.
This broadly simulates running in a mod_perl environment without using shared memory.
An example:
sub benchmark_functions_for_memory_cache { my ( $self, $template_dir, $cache_dir ) = @_; my ( @template_dirs ); @template_dirs = ( $template_dir ); return( { HT => sub { my $t = HTML::Template->new( type => 'filename', path => \@template_dirs, source => $_[ 0 ], case_sensitive => 1, cache => 1, die_on_bad_params => 0, ); $t->param( $_[ 1 ] ); $t->param( $_[ 2 ] ); $t->output(); }, } ); }
As can be seen, on each invocation the same code is run, with no stateful information carried between invocations.
This cache type should be provided only if there is some degree of reuse of data-structures from a previous invocation, such as reusing a previously-created template instance, or a compiled subroutine reference, by the benchmark function itself.
For example, if the benchmark function instantiates a copy of the template engine on the first run, loading the filename given, and then stores that instance in a local variable, then on subsequent invocations reuses that instance rather than starting from the beginning each time.
Or, an instance is created, the template loaded and compiled to a subroutine reference, that reference is stored, and subsequent invocations resume from that point.
Some examples:
sub benchmark_functions_for_instance_reuse { my ( $self, $template_dir, $cache_dir ) = @_; my ( $tt, $tt_x, $tt_xcet, @template_dirs ); @template_dirs = ( $template_dir ); $tt = Template->new( STASH => Template::Stash->new(), INCLUDE_PATH => \@template_dirs, ); return( { TT => sub { my $out; $tt->process( $_[ 0 ], { %{$_[ 1 ]}, %{$_[ 2 ]} }, \$out ); $out || $tt->error(); }, } ); }
This example shows the Template::Toolkit instance being reused between invocations.
sub benchmark_functions_for_instance_reuse { my ( $self, $template_dir, $cache_dir ) = @_; my ( $t ); return( { TeMMHM => sub { $t = Text::MicroMason->new()->compile( file => File::Spec->catfile( $template_dir, $_[ 0 ] ) ) unless $t; $t->( ( %{$_[ 1 ]}, %{$_[ 2 ]} ) ); }, } ); }
And this example shows that an intermediate stage is preserved as a function reference on the first invocation, then subsequent invocations just invoke that reference.
This cache type simulates running in a mod_perl environment with some form of memory caching, but the end-user of the template system would need to write some DIY caching themselves, and the benchmark doesn't include the overhead of just what that caching might be.
Different template engines support different template features, so Template::Benchmark allows the person performing the benchmarks to mix-and-match the features they wish to benchmark.
Those template engines that support the feature will be benchmarked and the end-user will be informed of which template engines didn't support which features.
To this end, Template::Benchmark queries the plugin for the template syntax required to implement each feature. That is, to generate the correct output from the given template variables, in the correct manner.
To ensure that like-for-like comparisons are being made, there are several variants of some basic template features, aimed to reflect nuances of common use.
literal_text
A chunk of literal text, dumped through to the output largely unchanged from its form in the template. ("Largely unchanged" means unescaping backslashes or the equivilent is fine.)
The block of literal text to be used is:
foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo
As produced by:
join( "\n", ( join( ' ', ( 'foo' ) x 12 ) ) x 5 )
scalar_variable
Interpolation of a template variable named scalar_variable.
hash_variable_value
Interpolation of a template variable stored in the hashref named hash_variable with key 'hash_value_key'.
hash_variable
'hash_value_key'
array_variable_value
Interpolation of a template variable stored in the arrayref named array_variable with index 2.
array_variable
2
deep_data_structure_value
Interpolation of a template variable stored in the hashref named this with nested keys 'is', 'a', 'very', 'deep', 'hash', 'structure'.
this
'is'
'a'
'very'
'deep'
'hash'
'structure'
This feature is designed to stress the speed that the template engine traverses deep data-structures.
array_loop_value
array_loop_template
Loop through the arrayref template variable named array_loop, inserting each element into the template output in turn.
array_loop
No delimiter is expected so if array_loop had value
[ 'one', 'two', 'three' ]
the output would look like
onetwothree
The reason there's no delimiter between records is to keep the template simple and to avoid any situations where differing behaviour creeps in from different template engines: for example if a newline was output after each element, some template engines would insert (or trim) additional white space as part of the flow control block, and while there may be ways to configure that behaviour within the template engine, that constitutes doing additional work over that done by other engines and would skew the benchmark's result away from being just the cost of doing the loop.
The _value version of this template feature permits any method of generating the content containing the value from the array, whereas the _template version requires that the output be produced by a block of template.
_value
_template
An example of this distinction would be from Template::Benchmark::Engines::TextTemplate:
array_loop_value => '{ $OUT .= $_ foreach @array_loop; }',
While a loop can be executed in the embedded perl, it can only build a literal string to be inserted back into the template output, there is no way to say that the loop means 'repeat this section of template' like that allowed, for example, in Template::Toolkit:
array_loop_template => '[% FOREACH i IN array_loop %][% i %][% END %]',
While it may be possible to coerce some embedded perl examples to do something similar by creating a new template engine instance and running a template fragment on it, that falls into the realms of DIY solutions discussed in "Supported or Unsupported?"
This distinction is important because it determines how easy it is to have large repeated sections of template without having to fall back to generating them within perl (which is presumably what you were trying to avoid by using a template system in the first place.)
hash_loop_value
hash_loop_template
Loop through the hashref template variable named hash_loop, in alphabetic order of the keys, inserting each key and value into the template output.
hash_loop
The key and value should be seperated by ': ' but between key/value pairs there's no delimiter.
': '
{ 'one' => 1, 'two' => 2, 'three' => 3 }
would produce
one: 1three: 3two: 2
The _value and _template versions of this template feature follow the same rules as documented for array_loop_value and array_loop_template.
records_loop_value
records_loop_template
Loop across an arrayref of hashrefs, much like that returned from a DBI fetchall_arrayref( {} ), for each 'record' output the value of the 'name' and 'age' keys.
fetchall_arrayref( {} )
'name'
'age'
As with hash_loop_value, a ': ' seperates name from age, and no delimiter between records.
[ { name => 'Andy MacAndy', age => 12, }, { name => 'Joe Jones', age => 10, }, { name => 'Jenny Jenkins', age => 11, }, ]
would give
Andy MacAndy: 12Joe Jones: 10Jenny Jenkins: 11
constant_if_literal
constant_if_template
Conditionally choose to insert some content if a constant literal 1 is true.
In the case of constant_if_literal the content is the literal text 'true' and for constant_if_template the content is the result of a template block inserting the content of template variable template_if_true.
'true'
template_if_true
The distinction between _literal and _template versions of this test are similar to those between array_loop_value and array_loop_template: the _template version must result from executing a block of the template markup rather than perl string manipulation.
_literal
variable_if_literal
variable_if_template
Conditionally choose to insert some content if the template variable variable_if is true.
variable_if
In the case of variable_if_literal the content is the literal text 'true' and for variable_if_template the content is the result of a template block inserting the content of template variable template_if_true.
constant_if_else_literal
constant_if_else_template
Conditionally choose to insert some content if a constant literal 1 is true, or some other content if it's false.
In the case of constant_if_else_literal the content is the literal text 'true' for true, and 'false' for false, and for constant_if_else_template the content is the result of a template block inserting the content of template variable template_if_true if true, or template_if_false if false.
'false'
template_if_false
variable_if_else_literal
variable_if_else_template
Conditionally choose to insert some content if the template variable variable_if_else is true, or some other content if it's false.
variable_if_else
In the case of variable_if_else_literal the content is the literal text 'true' for true, and 'false' for false, and for variable_if_else_template the content is the result of a template block inserting the content of template variable template_if_true if true, or template_if_false if false.
constant_expression
Insert the result of the constant expression 10 + 12.
10 + 12
Note that the template should actually calculate this, don't just put a literal 22 in the template, as the purpose of this feature is to determine if constants are subjected to constant-folding optimizations by the template engine and to give some indication of what gains are made by the engine in that situation.
22
variable_expression
Insert the result of multiplying the template variables variable_expression_a and variable_expression_b, ie doing:
variable_expression_a
variable_expression_b
variable_expression_a * variable_expression_b
complex_variable_expression
Insert the result of the following expression:
( ( variable_expression_a * variable_expression_b ) + variable_expression_a - variable_expression_b ) / variable_expression_b
Note that the brackets should be included, even if the template engine would sort out precedence correctly, because processing of brackets and precedence is part of what is being benchmarked by this feature.
Also note that the values of variable_expression_a and variable_expression_b are chosen so that the entire operation acts on integers and results in an integer value, so there is no need to worry about different floating-point precisions or output formats.
This feature is intended to be a slightly more stressful version of variable_expression, to allow comparision between the two results to isolate the expression engine performance of a template engine.
constant_function
Perform a function call (or equivilent, such as vmethod) within a template expression, on a constant literal.
The expression should do the equivilent of the perl:
substr( 'this has a substring', 11, 9 )
Like the difference between constant_expression and variable_expression this is to detect/benchmark any constant-folding optimizations.
variable_function
Perform a function call (or equivilent, such as vmethod) within a template expression, on the template variable variable_function_arg.
variable_function_arg
substr( $variable_function_arg, 4, 2 )
Sam Graham, <libtemplate-benchmark-perl at illusori.co.uk>
<libtemplate-benchmark-perl at illusori.co.uk>
Please report any bugs or feature requests to bug-template-benchmark at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Template-Benchmark. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-template-benchmark at rt.cpan.org
You can find documentation for this module with the perldoc command.
perldoc Template::Benchmark::Engine
You can also look for information at:
RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Template-Benchmark
AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Template-Benchmark
CPAN Ratings
http://cpanratings.perl.org/d/Template-Benchmark
Search CPAN
http://search.cpan.org/dist/Template-Benchmark/
Thanks to Paul Seamons for creating the the bench_various_templaters.pl script distributed with Template::Alloy, which was the ultimate inspiration for this module.
Copyright 2010 Sam Graham.
This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.
See http://dev.perl.org/licenses/ for more information.
To install Template::Benchmark, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Template::Benchmark
CPAN shell
perl -MCPAN -e shell install Template::Benchmark
For more information on module installation, please visit the detailed CPAN module installation guide.