The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.

Search results for "dist:WWW-RobotRules AnyDBM_File"

WWW::RobotRules::AnyDBM_File - Persistent RobotRules River stage four • 4 direct dependents • 6289 total dependents
6 ++

This is a subclass of *WWW::RobotRules* that uses the AnyDBM_File package to implement persistent diskcaching of robots.txt and host visit information. The constructor (the new() method) takes an extra argument specifying the name of the DBM file to ...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 UTC

WWW::RobotRules - database of robots.txt-derived permissions River stage four • 4 direct dependents • 6289 total dependents
6 ++

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The pars...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 UTC
2 results (0.041 seconds)