NAME

HTTP::RobotsTag - Parse Robots Tag In HTTP Headers

SYNOPSIS

  use HTTP::RobotsTag;

  my $response = $lwp->get( $url );
  my $p        = HTTP::RobotsTag->new();
  my $rule     = $p->parse_headers( $response );

  if ($rule->can_index()) {
    ...
  }

  if ($rule->is_available( $dt )) {
    ...
  }

DESCRIPTION

HTTP::RobotsTag parses HTTP headers for X-Robots-Tag headers and stores the information in a Rules object.

METHODS

new

parse_headers($headers)

Accepts a HTTP::Headers (or an object that implements the header()) method, and looks for X-Robots-Tag headers.

AUTHOR

Copyright (c) 2007 Daisuke Maki <daisuke@endeworks.jp>

SEE ALSO

HTML::RobotsTag::Rules

LICENSE

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See http://www.perl.com/perl/misc/Artistic.html