NAME

HTTP::RobotsTag::Rules - A Representation Of Robots Exclusion Rules

SYNOPSIS

  $rules = $p->parse_rules($html);
  $bool  = $rules->can_index();
  $bool  = $rules->can_follow();
  $bool  = $rules->can_archive();
  $bool  = $rules->can_snippet();
  $bool  = $rules->is_available( $dt );

DESCRIPTION

HTTP::RobotsTag::Rules represents the robots exclusion policies that are described within HTTP headers.

METHODS

new

can_index

can_follow

can_archive

can_snippet

is_available($dt)

Returns true if the resource is available on $dt.