The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

Search results for "dist:WWW-RobotRules robots.txt"

WWW::RobotRules - database of robots.txt-derived permissions River stage four • 4 direct dependents • 6202 total dependents
6 ++

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The pars...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 UTC

WWW::RobotRules::AnyDBM_File - Persistent RobotRules River stage four • 4 direct dependents • 6202 total dependents
6 ++

This is a subclass of *WWW::RobotRules* that uses the AnyDBM_File package to implement persistent diskcaching of robots.txt and host visit information. The constructor (the new() method) takes an extra argument specifying the name of the DBM file to ...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 UTC
2 results (0.024 seconds)