AnyEvent::HTTP::MultiGet - AnyEvent->condvar Control Freindly LWP Like agent
use Modern::Perl; use AnyEvent::HTTP::MultiGet; my $self=AnyEvent::HTTP::MultiGet->new(); my $count=0; TEST_LOOP: { my $req=HTTP::Request->new(GET=>'http://google.com'); my $req_b=HTTP::Request->new(GET=>'https://127.0.0.1:5888'); my @todo=HTTP::Request->new(GET=>'http://yahoo.com'); push @todo,HTTP::Request->new(GET=>'http://news.com'); push @todo,HTTP::Request->new(GET=>'https://news.com'); my $total=2 + scalar(@todo); my $cv=AnyEvent->condvar; my $code; $code=sub { my ($obj,$request,$result)=@_; printf 'HTTP Response code: %i'."\n",$result->code; ++$count; if(my $next=shift @todo) { $self->add_cb($req,$code); $self->run_next; } no warnings; $cv->send if $total==$count; }; $self->add_cb($req,$code); $self->add_cb($req_b,$code); $self->run_next; $cv->recv; }
Handling Multiple large http requests at once
use Modern::Perl; use AnyEvent::HTTP::MultiGet; my $self=AnyEvent::HTTP::MultiGet->new(); my $chunks=0; my $count=0; my $req=HTTP::Request->new(GET=>'https://google.com'); my $req_b=HTTP::Request->new(GET=>'https://yahoo.com'); my $req_c=HTTP::Request->new(GET=>'https://news.com'); $total=3; my @todo; push @todo,HTTP::Request->new(GET=>'https://127.0.0.1:5888'); push @todo,HTTP::Request->new(GET=>'https://127.0.0.1:5887'); push @todo,HTTP::Request->new(GET=>'https://127.0.0.1:5886'); push @todo,HTTP::Request->new(GET=>'https://127.0.0.1:5885'); $total +=scalar(@todo); TEST_LOOP: { my $on_body=sub { my ($getter,$request,$headers,$chunk)=@_; # 0: Our AnyEvent::HTTP::MultiGet instance # 1: the HTTP::Request object # 2: An HTTP::Headers object representing the current headers # 3: Current Data Chunk ++$chunks; printf 'request is %s'."\n",$request->uri; printf 'status code was: %i'."\n",$headers->header('Status'); printf 'content length was: %i'."\n",length($body); }; my $code; my $cb=AnyEvent->condvar; $code=sub { my ($obj,$request,$result)=@_; printf 'HTTP Response code: %i %s'."\n",$result->code,$request->url; ++$count; print "We are at response $count\n"; if(my $next=shift @todo) { $self->add_cb([$next,on_body=>$on_body],$code); $self->run_next; } no warnings; $cv->send if $count==$total; }; $self->add_cb([$req,on_body=>$on_body],$code); $self->add_cb([$req_b,on_body=>$on_body],$code); $self->add_cb([$req_c,on_body=>$on_body],$code); $self->run_next; $cv->recv; }
This class provides an AnyEvent->condvar frienddly implementation of HTTP::MultiGet.
Arguemnts and object accessors:
logger: DOES(Log::Log4perl::Logger) request_opts: See AnyEvent::HTTP params for details timeout: Global timeout for everything max_que_count: How many requests to run at once max_retry: How many times to retry if we get a connection/negotiation error
For internal use only:
in_control_loop: true when in the control loop stack: Data::Queue object que_count: Total Number of elements active in the que retry: Anonymous hash used to map ids to retry counts cb_map: Anonymous hash used to map ids to callbacks
my $id=$self->add_cb($request,$code)
Adds a request with a callback handler.
my $id=$self->add_cb([$request,key=>value],$code);
Wrapping [$request] allows passing additional key value to AnyEvent::HTTP::Request, with one exception, on_body=>$code is wrapped an additional callback.
Michael Shipper <AKALINUX@CPAN.ORG>
To install HTTP::MultiGet, copy and paste the appropriate command in to your terminal.
cpanm
cpanm HTTP::MultiGet
CPAN shell
perl -MCPAN -e shell install HTTP::MultiGet
For more information on module installation, please visit the detailed CPAN module installation guide.