Search results for "module:LWP::RobotUA"
LWP::RobotUA - a class for well-behaved Web robots
This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently. B...
OALDERS/libwww-perl-6.77 - 11 Mar 2024 00:58:25 UTC
LWP::Parallel::RobotUA - A class for Parallel Web Robots
This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently. B...
MSOUTH/ParallelUserAgent-2.62 - 29 May 2016 18:55:38 UTC
WWW::Scraper - framework for scraping results from search engines.
NOTE: You can find a full description of the Scraper framework in WWW::Scraper::ScraperPOD.pm. "Scraper" is a framework for issuing queries to a search engine, and scraping the data from the resultant multi-page responses, and the associated detail p...
GLENNWOOD/Scraper-3.05 - 02 Aug 2003 07:47:12 UTC
LWP::RobotPUA - Parallel LWP::RobotUA
RobotPUA is a simple frontend to the LWP::Parallel::RobotUA module. It is here in order to maintain the compatibility with previous releases. However, in order to prevent the previous need for changing the original LWP sources, all extension files ha...
MSOUTH/ParallelUserAgent-2.62 - 29 May 2016 18:55:38 UTC
LWP - The World-Wide Web library for Perl
The libwww-perl collection is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web. The main focus of the library is to provide classes and functions that allow you to write WWW cl...
OALDERS/libwww-perl-6.77 - 11 Mar 2024 00:58:25 UTC
LWP::Parallel - Extension for LWP to allow parallel HTTP and FTP access
Introduction ParallelUserAgent is an extension to the existing libwww module. It allows you to take a list of URLs (it currently supports HTTP, FTP, and FILE URLs. HTTPS might work, too) and connect to all of them _in parallel_, then wait for the res...
MSOUTH/ParallelUserAgent-2.62 - 29 May 2016 18:55:38 UTC
LWP::FastRobot - a robot which lets us check how long it will wait
Just like a RobotUA, but you can see how long it will wait for a given link and then try another....
MIKEDLR/Link_Controller-0.037 - 09 Feb 2002 18:12:34 UTC
LWP::NoStopRobot - a robot doesn't stop, but remembers where it has been
This module implements a user agent which remembers where it has been and when so that the user can avoid too fast visits, but doesn't actual implement that wait....
MIKEDLR/Link_Controller-0.037 - 09 Feb 2002 18:12:34 UTC