Search results for "dist:libwww-perl robots.txt"
LWP - The World-Wide Web library for Perl
The libwww-perl collection is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web. The main focus of the library is to provide classes and functions that allow you to write WWW cl...
OALDERS/libwww-perl-6.77 - 11 Mar 2024 00:58:25 UTC
lwptut - An LWP Tutorial
LWP (short for "Library for WWW in Perl") is a very popular group of Perl modules for accessing data on the Web. Like most Perl module-distributions, each of LWP's component modules comes with documentation that is a complete reference to its interfa...
OALDERS/libwww-perl-6.77 - 11 Mar 2024 00:58:25 UTC
LWP::RobotUA - a class for well-behaved Web robots
This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently. B...
OALDERS/libwww-perl-6.77 - 11 Mar 2024 00:58:25 UTC