View on
MetaCPAN is shutting down
For details read Perl NOC. After June 25th this page will redirect to
Curtis "Ovid" Poe > HTML-TokeParser-Simple > HTML::TokeParser::Simple



Annotate this POD


Open  0
View/Report Bugs
Module Version: 3.16   Source  


HTML::TokeParser::Simple - Easy to use HTML::TokeParser interface


 use HTML::TokeParser::Simple;
 my $p = HTML::TokeParser::Simple->new( $somefile );

 while ( my $token = $p->get_token ) {
     # This prints all text in an HTML doc (i.e., it strips the HTML)
     next unless $token->is_text;
     print $token->as_is;


HTML::TokeParser is an excellent module that's often used for parsing HTML. However, the tokens returned are not exactly intuitive to parse:

 ["S",  $tag, $attr, $attrseq, $text]
 ["E",  $tag, $text]
 ["T",  $text, $is_data]
 ["C",  $text]
 ["D",  $text]
 ["PI", $token0, $text]

To simplify this, HTML::TokeParser::Simple allows the user ask more intuitive (read: more self-documenting) questions about the tokens returned.

You can also rebuild some tags on the fly. Frequently, the attributes associated with start tags need to be altered, added to, or deleted. This functionality is built in.

Since this is a subclass of HTML::TokeParser, all HTML::TokeParser methods are available. To truly appreciate the power of this module, please read the documentation for HTML::TokeParser and HTML::Parser.



The constructor for HTML::TokeParser::Simple can be used just like HTML::TokeParser's constructor:

  my $parser = HTML::TokeParser::Simple->new($filename);
  # or
  my $parser = HTML::TokeParser::Simple->new($filehandle);
  # or
  my $parser = HTML::TokeParser::Simple->new(\$html_string);

new($source_type, $source)

If you wish to be more explicit, there is a new style of constructor available.

  my $parser = HTML::TokeParser::Simple->new(file   => $filename);
  # or
  my $parser = HTML::TokeParser::Simple->new(handle => $filehandle);
  # or
  my $parser = HTML::TokeParser::Simple->new(string => $html_string);

Note that you do not have to provide a reference for the string if using the string constructor.

As a convenience, you can also attempt to fetch the HTML directly from a URL.

  my $parser = HTML::TokeParser::Simple->new(url => 'http://some.url');

This method relies on LWP::Simple. If this module is not found or the page cannot be fetched, the constructor will croak().



This method will return the next token that HTML::TokeParser::get_token() method would return. However, it will be blessed into a class appropriate which represents the token type.


This method will return the next token that HTML::TokeParser::get_tag() method would return. However, it will be blessed into either the HTML::TokeParser::Simple::Token::Tag::Start or HTML::TokeParser::Simple::Token::Tag::End class.


As of version 3.14, you can now peek() at the upcomings tokens without affecting the state of the parser. By default, peek() will return the text of the next token, but specifying an integer $count will return the text of the next $count tokens.

This is useful when you're trying to debug where you are in a document.

 warn $parser->peek(3); # show the next 3 tokens


The following methods may be called on the token object which is returned, not on the parser object.

Boolean Accessors

These accessors return true or false.

Data Accessors

Some of these were originally return_ methods, but that name was not only unwieldy, but also went against reasonable conventions. The get_ methods listed below still have return_ methods available for backwards compatibility reasons, but they merely call their get_ counterpart. For example, calling return_tag() actually calls get_tag() internally.


The delete_attr() and set_attr() methods allow the programmer to rewrite start tag attributes on the fly. It should be noted that bad HTML will be "corrected" by this. Specifically, the new tag will have all attributes lower-cased with the values properly quoted.

Self-closing tags (e.g. <hr />) are also handled correctly. Some older browsers require a space prior to the final slash in a self-closed tag. If such a space is detected in the original HTML, it will be preserved.

Calling a mutator on an token type that does not support that property is a no-op. For example:

 if ($token->is_comment) {
    $token->set_attr(foo => 'bar'); # does nothing


The parser returns tokens that are blessed into appropriate classes. Some people get confused and try to call parser methods on tokens and token methods on the parser. To prevent this, HTML::TokeParser::Simple versions 1.4 and above now bless all tokens into appropriate token classes. Please keep this in mind while using this module (and many thanks to PodMaster for pointing out this issue to me.)


Finding comments

For some strange reason, your Pointy-Haired Boss (PHB) is convinced that the graphics department is making fun of him by embedding rude things about him in HTML comments. You need to get all HTML comments from the HTML.

 use strict;
 use HTML::TokeParser::Simple;

 my @html_docs = glob( "*.html" );

 open PHB, "> phbreport.txt" or die "Cannot open phbreport for writing: $!";

 foreach my $doc ( @html_docs ) {
     print "Processing $doc\n";
     my $p = HTML::TokeParser::Simple->new( file => $doc );
     while ( my $token = $p->get_token ) {
         next unless $token->is_comment;
         print PHB $token->as_is, "\n";

 close PHB;

Stripping Comments

Uh oh. Turns out that your PHB was right for a change. Many of the comments in the HTML weren't very polite. Since your entire graphics department was just fired, it falls on you need to strip those comments from the HTML.

 use strict;
 use HTML::TokeParser::Simple;

 my $new_folder = 'no_comment/';
 my @html_docs  = glob( "*.html" );

 foreach my $doc ( @html_docs ) {
     print "Processing $doc\n";
     my $new_file = "$new_folder$doc";

     open PHB, "> $new_file" or die "Cannot open $new_file for writing: $!";

     my $p = HTML::TokeParser::Simple->new( $file => doc );
     while ( my $token = $p->get_token ) {
         next if $token->is_comment;
         print PHB $token->as_is;
     close PHB;

Changing form tags

Your company was and now is Unfortunately, whoever wrote your HTML decided to hardcode "" into the action attribute of the form tags. You need to change it to "".

 use strict;
 use HTML::TokeParser::Simple;

 my $new_folder = 'new_html/';
 my @html_docs  = glob( "*.html" );

 foreach my $doc ( @html_docs ) {
     print "Processing $doc\n";
     my $new_file = "$new_folder$doc";

     open FILE, "> $new_file" or die "Cannot open $new_file for writing: $!";

     my $p = HTML::TokeParser::Simple->new( file => $doc );
     while ( my $token = $p->get_token ) {
         if ( $token->is_start_tag('form') ) {
             my $action = $token->get_attr(action);
             $action =~ s/www\.foo\.com/;
             $token->set_attr('action', $action);
         print FILE $token->as_is;
     close FILE;


For compatibility reasons with HTML::TokeParser, methods that return references are violating encapsulation and altering the references directly will alter the state of the object. Subsequent calls to rewrite_tag() can thus have unexpected results. Do not alter these references directly unless you are following behavior described in these docs. In the future, certain methods such as get_attr, get_attrseq and others may return a copy of the reference rather than the original reference. This behavior has not yet been changed in order to maintain compatibility with previous versions of this module. At the present time, your author is not aware of anyone taking advantage of this "feature," but it's better to be safe than sorry.

Use of $HTML::Parser::VERSION which is less than 3.25 may result in incorrect behavior as older versions do not always handle XHTML correctly. It is the programmer's responsibility to verify that the behavior of this code matches the programmer's needs.

Note that HTML::Parser processes text in 512 byte chunks. This sometimes will cause strange behavior and cause text to be broken into more than one token. You can suppress this behavior with the following command:

 $p->unbroken_text( [$bool] );

See the HTML::Parser documentation and for more information.


There are no known bugs, but that's no guarantee.

Address bug reports and comments to: <>. When sending bug reports, please provide the version of HTML::Parser, HTML::TokeParser, HTML::TokeParser::Simple, the version of Perl, and the version of the operating system you are using.

Reverse the name to email the author.


You may wish to change the behavior of this module. You probably do not want to subclass HTML::TokeParser::Simple. Instead, you'll want to subclass one of the token classes. HTML::TokeParser::Simple::Token is the base class for all tokens. Global behavioral changes should go there. Otherwise, see the appropriate token class for the behavior you wish to alter.









Copyright (c) 2004 by Curtis "Ovid" Poe. All rights reserved. This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself


Curtis "Ovid" Poe <>

Reverse the name to email the author.

syntax highlighting: