NAME

Dezi::Bot - web crawler

SYNOPSIS

 use Dezi::Bot;

 my $bot = Dezi::Bot->new(
 
    # give your bot a name
    name => 'dezibot',  
    
    # explicit object, instead of class+config
    spider => $spider_object,  
     
    # every crawled URI
    # passed to the $handler->handle() method
    handler_class => 'Dezi::Bot::Handler',
    
    # default
    spider_class => 'Dezi::Bot::Spider',
    
    # passed to spider_class->new()
    spider_config   => {
        agent      => 'dezibot ' . $Dezi::Bot::VERSION,
        email      => 'bot@dezi.org',
        max_depth  => 4,
    },
    
    # default
    cache_class => 'Dezi::Bot::Cache',
    
    # passed to cache_class->new()
    cache_config => {
        driver      => 'File',
        root_dir    => '/tmp/dezibot',
    },
    
    # default
    queue_class => 'Dezi::Bot::Queue',
    
    # passed to queue_class->new()
    queue_config => {
        type     => 'DBI',
        dsn      => "DBI:mysql:database=dezibot;host=localhost;port=3306",
        username => 'myuser',
        password => 'mysecret',
    },
 );
 
 $bot->crawl('http://dezi.org');

DESCRIPTION

The Dezi::Bot module is a web crawler optimized for parallel use across multiple hosts.

METHODS

init( args )

Overrides the base method to set default options based on args. See the SYNOPSIS.

Options:

name

spider

handler_class

handler_config

spider_class

spider_config

cache_class

cache_config

queue_class

queue_config

crawl( urls )

Calls ->spider->crawl() for an array of urls.

Returns the total number of URIs crawled.

AUTHOR

Peter Karman, <karman at cpan.org>

BUGS

Please report any bugs or feature requests to bug-dezi-bot at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Dezi-Bot. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

    perldoc Dezi::Bot

You can also look for information at:

• RT: CPAN's request tracker

  http://rt.cpan.org/NoAuth/Bugs.html?Dist=Dezi-Bot

• AnnoCPAN: Annotated CPAN documentation

  http://annocpan.org/dist/Dezi-Bot

• CPAN Ratings

  http://cpanratings.perl.org/d/Dezi-Bot

• Search CPAN

  http://search.cpan.org/dist/Dezi-Bot/

COPYRIGHT & LICENSE

Copyright 2013 Peter Karman.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.