The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

Search results for "dist:ParallelUserAgent robots.txt"

LWP::Parallel - Extension for LWP to allow parallel HTTP and FTP access River stage one • 2 direct dependents • 3 total dependents
2 ++

Introduction ParallelUserAgent is an extension to the existing libwww module. It allows you to take a list of URLs (it currently supports HTTP, FTP, and FILE URLs. HTTPS might work, too) and connect to all of them _in parallel_, then wait for the res...

MSOUTH/ParallelUserAgent-2.62 - 29 May 2016 18:55:38 UTC

LWP::Parallel::RobotUA - A class for Parallel Web Robots River stage one • 2 direct dependents • 3 total dependents
2 ++

This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently. B...

MSOUTH/ParallelUserAgent-2.62 - 29 May 2016 18:55:38 UTC
2 results (0.022 seconds)