Ralf S. Engelschall > lcwa > LWP::RobotUA

Download:
lcwa-1.0.0.tar.gz

Dependencies

Annotate this POD

Related Modules

LWP::Simple
WWW::Robot
URI::URL
LWP::UserAgent
WWW::Mechanize
HTML::LinkExtor
Data::Dumper
HTTP::Response
HTML::TokeParser
HTML::Parser
more...
By perlmonks.org

CPAN RT

New  3
Open  0
View/Report Bugs
Source  

NAME ^

LWP::RobotUA - A class for Web Robots

SYNOPSIS ^

  require LWP::RobotUA;
  $ua = new LWP::RobotUA 'my-robot/0.1', 'me@foo.com';
  $ua->delay(10);  # be very nice, go slowly
  ...
  # just use it just like a normal LWP::UserAgent
  $res = $ua->request($req);

DESCRIPTION ^

This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the robots.txt file to ensure that they are welcomed and they should not send too frequent requests.

But, before you consider writing a robot take a look at <URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.

When you use a LWP::RobotUA as your user agent, then you do not really have to think about these things yourself. Just send requests as you do when you are using a normal LWP::UserAgent and this special agent will make sure you are nice.

METHODS ^

The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the same methods. The use_alarm() method also desides whether we will wait if a request is tried too early (if true), or will return an error response (if false).

In addition these methods are provided:

$ua = LWP::RobotUA->new($agent_name, $from, [$rules])

Your robot's name and the mail address of the human responsible for the robot (i.e. you) is required by the constructor.

Optionally it allows you to specify the WWW::RobotRules object to use.

$ua->delay([$minutes])

Set the minimum delay between requests to the same server. The default is 1 minute.

$ua->rules([$rules])

Set/get which WWW::RobotRules object to use.

$ua->no_visits($netloc)

Returns the number of documents fetched from this server host.

$ua->host_wait($netloc)

Returns the number of seconds you must wait before you can make a new request to this host.

$ua->as_string

Returns a text that describe the state of the UA. Mainly useful for debugging.

SEE ALSO ^

LWP::UserAgent, WWW::RobotRules

AUTHOR ^

Gisle Aas <aas@sn.no>

syntax highlighting: