Data::Stream::Bulk - N at a time iteration API
# get a bulk stream from somewere
my $s = Data::Stream::Bulk::Foo->new( ... );
# can be used like this:
until ( $s->is_done ) {
foreach my $item ( $s->items ) {
process($item);
}
}
# or like this:
while( my $block = $s->next ) {
foreach my $item ( @$block ) {
process($item);
}
}
This module tries to find middle ground between one at a time and all at once processing of data sets.
The purpose of this module is to avoid the overhead of implementing an iterative api when this isn't necessary, without breaking forward compatibility in case that becomes necessary later on.
The API optimizes for when a data set typically fits in memory and is returned as an array, but the consumer cannot assume that the data set is bounded.
The API is destructive in order to minimize the chance that resultsets are leaked due to improper usage.
The API requires two methods to be implemented:
Should return true if the stream is exhausted.
As long as this method returns a false value (not done) next could potentially return another block.
Returns the next block.
Note that next is not guaranteed to return an array reference, even if is_done returned false prior to calling it.
This method calls next and dereferences the result if there are pending items.
Force evaluation of the entire resultset.
Note that for large data sets this might cause swap thrashing of various other undesired effects. Use with caution.
Concatenates this stream with @streams, returning a single stream.
Returns a possibly cleaned up list of streams.
Used by cat.
Overridden by Data::Stream::Bulk::Array, Data::Stream::Bulk::Cat and Data::Stream::Bulk::Nil to implement some simple short circuiting.
Applies a per-block block filter to the stream.
Returns a possibly new stream with the filtering layered.
$filter is invoked once per block and should return an array reference to the filtered block.
Should be overridden to return true if all the items are already realized (e.g. in the case of Data::Stream::Bulk::Array).
Returns false by default.
When true calling all is supposed to be safe (memory usage should be in the same order of magnitude as stream's own usage).
This is typically useful when tranforming an array is easier than transorming a stream (e.g. optional duplicate filtering).
This class is not a stream at all, but just one block. When the data set easily fits in memory this class can be used, while retaining forward compatibility with larger data sets.
Callback driven iteration.
Bulk fetching of data from DBI statement handles.
DBIx::Class::ResultSet iteration.
An empty result set.
A concatenation of several streams.
A filter wrapping a stream.
HOP::Stream, Iterator, Class::Iterator etc for one by one iteration
http://en.wikipedia.org/wiki/MapReduce, LISP, and all that other kool aid
Add a hint for sorted streams (like loaded but as an attribute in the base role).
Introduce a merge operation for merging of sorted streams.
Optimize unique to make use of sorting hints for constant space uniquing.
To assist in proccessing and creating streams.
This module is maintained using Darcs. You can get the latest version from http://nothingmuch.woobling.org/code, and use darcs send to commit changes.
Yuval Kogman <nothingmuch@woobling.org>
Copyright (c) 2008 Yuval Kogman. All rights reserved
This program is free software; you can redistribute
it and/or modify it under the same terms as Perl itself.
