Search results for "dist:Dezi-Bot KARMAN"
Dezi::Bot - web crawler
The Dezi::Bot module is a web crawler optimized for parallel use across multiple hosts....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Cache - web crawler cache
The Dezi::Bot::Cache module conforms to the SWISH::Prog::Cache API but delegates all caching to CHI....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Utils - web crawler utilities
The Dezi::Bot::Utils provides utility functions for Dezi::Bot....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Spider - web spider
The Dezi::Bot::Spider is a subclass of SWISH::Prog::Aggregator::Spider....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Queue - web crawler queue
The Dezi::Bot::Queue module adheres to the API of SWISH::Prog::Queue while optimized for persistent storage....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Handler - web crawler handler
The Dezi::Bot::Handler manages each doc the crawler successfully encounters....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Queue::DBI - web crawler queue with DBI storage
The Dezi::Bot::Queue::DBI class is a subclass of Dezi::Bot::Queue that uses DBI for storage....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
Dezi::Bot::Handler::FileCacher - web crawler handler that caches files
The Dezi::Bot::Handler::FileCacher writes each doc to the filesystem, managing its progress and status via DBI....
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC
dezibot - parallel web crawler
dezibot is a command line tool wrapping the Dezi::Bot module. dezibot can: * read from a config file or take options on the command line * read URLs from a file or from @ARGV * spawn multiple parallel spiders...
KARMAN/Dezi-Bot-0.003 - 01 May 2014 03:23:29 UTC