The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.

Search results for "dist:WWW-RobotRules MARTIJN"

WWW::RobotRules - database of robots.txt-derived permissions River stage four • 4 direct dependents • 6289 total dependents

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The pars...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 UTC
1 result (0.021 seconds)