Daisuke Maki > Gungho > Gungho::Component::RobotRules

Download:
Gungho-0.09008.tar.gz

Dependencies

Annotate this POD

CPAN RT

New  6
Open  4
View/Report Bugs
Source  

NAME ^

Gungho::Component::RobotRules - Respect robots.txt

SYNOPSIS ^

  ---
  components:
    - RobotRules

METHODS ^

setup

setup_robot_rules_parser

setup_robot_rules_storage

handle_response

send_request

allowed($request)

Returns 1 if request is allowed to be fetched, 0 if not. -1 and -2 are returned when there is a pending request to fetch /robots.txt

dispatch_pending_robots_txt

Dispatches requests that were pending because of a missing robots.txt entry

push_pending_robots_txt

Pushes a request in the wait queue for a robots.txt

parse_robot_rules

Parses the robot rule and stores it

syntax highlighting: