TAP::Parser - Parse TAP output
Version 3.00
use TAP::Parser; my $parser = TAP::Parser->new( { source => $source } ); while ( my $result = $parser->next ) { print $result->as_string; }
TAP::Parser is designed to produce a proper parse of TAP output. For an example of how to run tests through this module, see the simple harnesses examples/.
TAP::Parser
examples/
There's a wiki dedicated to the Test Anything Protocol:
http://testanything.org
It includes the TAP::Parser Cookbook:
http://testanything.org/wiki/index.php/TAP::Parser_Cookbook
new
my $parser = TAP::Parser->new(\%args);
Returns a new TAP::Parser object.
The arguments should be a hashref with one of the following keys:
source
This is the preferred method of passing arguments to the constructor. To determine how to handle the source, the following steps are taken.
If the source contains a newline, it's assumed to be a string of raw TAP output.
If the source is a reference, it's assumed to be something to pass to the TAP::Parser::Iterator::Stream constructor. This is used internally and you should not use it.
Otherwise, the parser does a -e check to see if the source exists. If so, it attempts to execute the source and read the output as a stream. This is by far the preferred method of using the parser.
-e
foreach my $file ( @test_files ) { my $parser = TAP::Parser->new( { source => $file } ); # do stuff with the parser }
tap
The value should be the complete TAP output.
exec
If passed an array reference, will attempt to create the iterator by passing a TAP::Parser::Source object to TAP::Parser::Iterator::Source, using the array reference strings as the command arguments to IPC::Open3::open3:
exec => [ '/usr/bin/ruby', 't/my_test.rb' ]
Note that source and exec are mutually exclusive.
The following keys are optional.
callback
If present, each callback corresponding to a given result type will be called with the result as the argument if the run method is used:
run
my %callbacks = ( test => \&test_callback, plan => \&plan_callback, comment => \&comment_callback, bailout => \&bailout_callback, unknown => \&unknown_callback, ); my $aggregator = TAP::Parser::Aggregator->new; foreach my $file ( @test_files ) { my $parser = TAP::Parser->new( { source => $file, callbacks => \%callbacks, } ); $parser->run; $aggregator->add( $file, $parser ); }
switches
If using a Perl file as a source, optional switches may be passed which will be used when invoking the perl executable.
my $parser = TAP::Parser->new( { source => $test_file, switches => '-Ilib', } );
spool
If passed a filehandle will write a copy of all parsed TAP to that handle.
merge
If false, STDERR is not captured (though it is 'relayed' to keep it somewhat synchronized with STDOUT.)
If true, STDERR and STDOUT are the same filehandle. This may cause breakage if STDERR contains anything resembling TAP format, but does allow exact synchronization.
Subtleties of this behavior may be platform-dependent and may change in the future.
next
my $parser = TAP::Parser->new( { source => $file } ); while ( my $result = $parser->next ) { print $result->as_string, "\n"; }
This method returns the results of the parsing, one result at a time. Note that it is destructive. You can't rewind and examine previous results.
If callbacks are used, they will be issued before this call returns.
Each result returned is a subclass of TAP::Parser::Result. See that module and related classes for more information on how to use them.
$parser->run;
This method merely runs the parser and parses all of the TAP.
If you've read this far in the docs, you've seen this:
while ( my $result = $parser->next ) { print $result->as_string; }
Each result returned is a TAP::Parser::Result subclass, referred to as result types.
Basically, you fetch individual results from the TAP. The six types, with examples of each, are as follows:
Version
TAP version 12
Plan
1..42
Test
ok 3 - We should start with some foobar!
Comment
# Hope we don't use up the foobar.
Bailout
Bail out! We ran out of foobar!
Unknown
... yo, this ain't TAP! ...
Each result fetched is a result object of a different type. There are common methods to each result object and different types may have methods unique to their type. Sometimes a type method may be overridden in a subclass, but its use is guaranteed to be identical.
type
Returns the type of result, such as comment or test.
comment
test
as_string
Prints a string representation of the token. This might not be the exact output, however. Tests will have test numbers added if not present, TODO and SKIP directives will be capitalized and, in general, things will be cleaned up. If you need the original text for the token, see the raw method.
raw
Returns the original line of text which was parsed.
is_plan
Indicates whether or not this is the test plan line.
is_test
Indicates whether or not this is a test line.
is_comment
Indicates whether or not this is a comment. Comments will generally only appear in the TAP stream if STDERR is merged to STDOUT. See the merge option.
is_bailout
Indicates whether or not this is bailout line.
is_yaml
Indicates whether or not the current item is a YAML block.
is_unknown
Indicates whether or not the current line could be parsed.
is_ok
if ( $result->is_ok ) { ... }
Reports whether or not a given result has passed. Anything which is not a test result returns true. This is merely provided as a convenient shortcut which allows you to do this:
my $parser = TAP::Parser->new( { source => $source } ); while ( my $result = $parser->next ) { # only print failing results print $result->as_string unless $result->is_ok; }
plan
if ( $result->is_plan ) { ... }
If the above evaluates as true, the following methods will be available on the $result object.
$result
if ( $result->is_plan ) { print $result->plan; }
This is merely a synonym for as_string.
tests_planned
my $planned = $result->tests_planned;
Returns the number of tests planned. For example, a plan of 1..17 will cause this method to return '17'.
1..17
directive
my $directive = $result->directive;
If a SKIP directive is included with the plan, this method will return it.
1..0 # SKIP: why bother?
explanation
my $explanation = $result->explanation;
If a SKIP directive was included with the plan, this method will return the explanation, if any.
commment
if ( $result->is_comment ) { ... }
if ( $result->is_comment ) { my $comment = $result->comment; print "I have something to say: $comment"; }
bailout
if ( $result->is_bailout ) { ... }
if ( $result->is_bailout ) { my $explanation = $result->explanation; print "We bailed out because ($explanation)"; }
If, and only if, a token is a bailout token, you can get an "explanation" via this method. The explanation is the text after the mystical "Bail out!" words which appear in the tap output.
unknown
if ( $result->is_unknown ) { ... }
There are no unique methods for unknown results.
if ( $result->is_test ) { ... }
ok
my $ok = $result->ok;
Returns the literal text of the ok or not ok status.
not ok
number
my $test_number = $result->number;
Returns the number of the test, even if the original TAP output did not supply that number.
description
my $description = $result->description;
Returns the description of the test, if any. This is the portion after the test number but before the directive.
Returns either TODO or SKIP if either directive was present for a test line.
TODO
SKIP
If a test had either a TODO or SKIP directive, this method will return the accompanying explantion, if present.
not ok 17 - 'Pigs can fly' # TODO not enough acid
For the above line, the explanation is not enough acid.
Returns a boolean value indicating whether or not the test passed. Remember that for TODO tests, the test always passes.
Note: this was formerly passed. The latter method is deprecated and will issue a warning.
passed
is_actual_ok
if ( $result->is_actual_ok ) { ... }
Returns a boolean value indicating whether or not the test passed, regardless of its TODO status.
Note: this was formerly actual_passed. The latter method is deprecated and will issue a warning.
actual_passed
is_unplanned
if ( $test->is_unplanned ) { ... }
If a test number is greater than the number of planned tests, this method will return true. Unplanned tests will always return false for is_ok, regardless of whether or not the test has_todo (see TAP::Parser::Result::Test for more information about this).
has_todo
has_skip
if ( $result->has_skip ) { ... }
Returns a boolean value indicating whether or not this test had a SKIP directive.
if ( $result->has_todo ) { ... }
Returns a boolean value indicating whether or not this test had a TODO directive.
Note that TODO tests always pass. If you need to know whether or not they really passed, check the is_actual_ok method.
in_todo
if ( $parser->in_todo ) { ... }
True while the most recent result was a TODO. Becomes true before the TODO result is returned and stays true until just before the next non- TODO test is returned.
After parsing the TAP, there are many methods available to let you dig through the results and determine what is meaningful to you.
These results refer to individual tests which are run.
my @passed = $parser->passed; # the test numbers which passed my $passed = $parser->passed; # the number of tests which passed
This method lets you know which (or how many) tests passed. If a test failed but had a TODO directive, it will be counted as a passed test.
failed
my @failed = $parser->failed; # the test numbers which failed my $failed = $parser->failed; # the number of tests which failed
This method lets you know which (or how many) tests failed. If a test passed but had a TODO directive, it will NOT be counted as a failed test.
# the test numbers which actually passed my @actual_passed = $parser->actual_passed; # the number of tests which actually passed my $actual_passed = $parser->actual_passed;
This method lets you know which (or how many) tests actually passed, regardless of whether or not a TODO directive was found.
actual_ok
This method is a synonym for actual_passed.
actual_failed
# the test numbers which actually failed my @actual_failed = $parser->actual_failed; # the number of tests which actually failed my $actual_failed = $parser->actual_failed;
This method lets you know which (or how many) tests actually failed, regardless of whether or not a TODO directive was found.
todo
my @todo = $parser->todo; # the test numbers with todo directives my $todo = $parser->todo; # the number of tests with todo directives
This method lets you know which (or how many) tests had TODO directives.
todo_passed
# the test numbers which unexpectedly succeeded my @todo_passed = $parser->todo_passed; # the number of tests which unexpectedly succeeded my $todo_passed = $parser->todo_passed;
This method lets you know which (or how many) tests actually passed but were declared as "TODO" tests.
todo_failed
# deprecated in favor of 'todo_passed'. This method was horribly misnamed.
This was a badly misnamed method. It indicates which TODO tests unexpectedly succeeded. Will now issue a warning and call todo_passed.
skipped
my @skipped = $parser->skipped; # the test numbers with SKIP directives my $skipped = $parser->skipped; # the number of tests with SKIP directives
This method lets you know which (or how many) tests had SKIP directives.
These results are "meta" information about the total results of an individual test program.
my $plan = $parser->plan;
Returns the test plan, if found.
good_plan
Deprecated. Use is_good_plan instead.
is_good_plan
if ( $parser->is_good_plan ) { ... }
Returns a boolean value indicating whether or not the number of tests planned matches the number of tests run.
Note: this was formerly good_plan. The latter method is deprecated and will issue a warning.
And since we're on that subject ...
print $parser->tests_planned;
Returns the number of tests planned, according to the plan. For example, a plan of '1..17' will mean that 17 tests were planned.
tests_run
print $parser->tests_run;
Returns the number of tests which actually were run. Hopefully this will match the number of $parser->tests_planned.
$parser->tests_planned
skip_all
Returns a true value (actually the reason for skipping) if all tests were skipped.
start_time
Returns the time when the Parser was created.
end_time
Returns the time when the end of TAP input was seen.
has_problems
if ( $parser->has_problems ) { ... }
This is a 'catch-all' method which returns true if any tests have currently failed, any TODO tests unexpectedly succeeded, or any parse errors occurred.
version
$parser->version;
Once the parser is done, this will return the version number for the parsed TAP. Version numbers were introduced with TAP version 13 so if no version number is found version 12 is assumed.
exit
$parser->exit;
Once the parser is done, this will return the exit status. If the parser ran an executable, it returns the exit status of the executable.
wait
$parser->wait;
Once the parser is done, this will return the wait status. If the parser ran an executable, it returns the wait status of the executable. Otherwise, this mererely returns the exit status.
parse_errors
my @errors = $parser->parse_errors; # the parser errors my $errors = $parser->parse_errors; # the number of parser_errors
Fortunately, all TAP output is perfect. In the event that it is not, this method will return parser errors. Note that a junk line which the parser does not recognize is not an error. This allows this parser to handle future versions of TAP. The following are all TAP errors reported by the parser:
not
Misplaced plan
The plan (for example, '1..5'), must only come at the beginning or end of the TAP output.
No plan
Gotta have a plan!
More than one plan
1..3 ok 1 - input file opened not ok 2 - first line of the input valid # todo some data ok 3 read the rest of the file 1..3
Right. Very funny. Don't do that.
Test numbers out of sequence
1..3 ok 1 - input file opened not ok 2 - first line of the input valid # todo some data ok 2 read the rest of the file
That last test line above should have the number '3' instead of '2'.
Note that it's perfectly acceptable for some lines to have test numbers and others to not have them. However, when a test number is found, it must be in sequence. The following is also an error:
1..3 ok 1 - input file opened not ok - first line of the input valid # todo some data ok 2 read the rest of the file
But this is not:
1..3 ok - input file opened not ok - first line of the input valid # todo some data ok 3 read the rest of the file
get_select_handles
Get an a list of file handles which can be passed to select to determine the readiness of this parser.
select
delete_spool
Delete and return the spool.
my $fh = $parser->delete_spool;
As mentioned earlier, a "callback" key may be added to the TAP::Parser constructor. If present, each callback corresponding to a given result type will be called with the result as the argument if the run method is used. The callback is expected to be a subroutine reference (or anonymous subroutine) which is invoked with the parser result as its argument.
Callbacks may also be added like this:
$parser->callback( test => \&test_callback ); $parser->callback( plan => \&plan_callback );
The following keys allowed for callbacks. These keys are case-sensitive.
Invoked if $result->is_test returns true.
$result->is_test
Invoked if $result->is_version returns true.
$result->is_version
Invoked if $result->is_plan returns true.
$result->is_plan
Invoked if $result->is_comment returns true.
$result->is_comment
Invoked if $result->is_unknown returns true.
$result->is_unknown
yaml
Invoked if $result->is_yaml returns true.
$result->is_yaml
ELSE
If a result does not have a callback defined for it, this callback will be invoked. Thus, if all of the previous result types are specified as callbacks, this callback will never be invoked.
ALL
This callback will always be invoked and this will happen for each result after one of the above callbacks is invoked. For example, if Term::ANSIColor is loaded, you could use the following to color your test output:
my %callbacks = ( test => sub { my $test = shift; if ( $test->is_ok && not $test->directive ) { # normal passing test print color 'green'; } elsif ( !$test->is_ok ) { # even if it's TODO print color 'white on_red'; } elsif ( $test->has_skip ) { print color 'white on_blue'; } elsif ( $test->has_todo ) { print color 'white'; } }, ELSE => sub { # plan, comment, and so on (anything which isn't a test line) print color 'black on_white'; }, ALL => sub { # now print them print shift->as_string; print color 'reset'; print "\n"; }, );
EOF
Invoked when there are no more lines to be parsed. Since there is no accompanying TAP::Parser::Result object the TAP::Parser object is passed instead.
If you're looking for an EBNF grammar, see TAP::Parser::Grammar.
The Perl-QA list attempted to ensure backwards compatability with Test::Harness. However, there are some minor differences.
TODO plans
A little-known feature of Test::Harness is that it supported TODO lists in the plan:
1..2 todo 2 ok 1 - We have liftoff not ok 2 - Anti-gravity device activated
Under Test::Harness, test number 2 would pass because it was listed as a TODO test on the plan line. However, we are not aware of anyone actually using this feature and hard-coding test numbers is discouraged because it's very easy to add a test and break the test number sequence. This makes test suites very fragile. Instead, the following should be used:
1..2 ok 1 - We have liftoff not ok 2 - Anti-gravity device activated # TODO
'Missing' tests
It rarely happens, but sometimes a harness might encounter 'missing tests:
ok 1 ok 2 ok 15 ok 16 ok 17
Test::Harness would report tests 3-14 as having failed. For the TAP::Parser, these tests are not considered failed because they've never run. They're reported as parse failures (tests out of sequence).
All of the following have helped. Bug reports, patches, (im)moral support, or just words of encouragement have all been forthcoming.
Michael Schwern
Andy Lester
chromatic
GEOFFR
Shlomi Fish
Torsten Schoenfeld
Jerry Gay
Aristotle
Adam Kennedy
Yves Orton
Adrian Howard
Sean & Lil
Andreas J. Koenig
Florian Ragwitz
Corion
Mark Stosberg
Matt Kraai
Curtis "Ovid" Poe <ovid@cpan.org>
Andy Armstong <andy@hexten.net>
Eric Wilhelm @ <ewilhelm at cpan dot org>
Michael Peters <mpeters at plusthree dot com>
Leif Eriksen <leif dot eriksen at bigpond dot com>
Please report any bugs or feature requests to bug-tapx-parser@rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=TAP-Parser. We will be notified, and then you'll automatically be notified of progress on your bug as we make changes.
bug-tapx-parser@rt.cpan.org
Obviously, bugs which include patches are best. If you prefer, you can patch against bleed by via anonymous checkout of the latest version:
svn checkout http://svn.hexten.net/tapx
Copyright 2006 Curtis "Ovid" Poe, all rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Test::Harness, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Test::Harness
CPAN shell
perl -MCPAN -e shell install Test::Harness
For more information on module installation, please visit the detailed CPAN module installation guide.