Queue::Q::ReliableFIFO - FIFO queue keeping track of claimed items
use Queue::Q::ReliableFIFO::Redis; # or ::Perl or ... my $q = ... create object of chosen ReliableFIFO implementation... # producer: $q->enqueue_item([qw(my data structure)]); # rinse repeat... # consumer: my $item = $q->claim_item; # work with data... $q->mark_item_as_done($item); # Implementation dependent: somewhere in a recovery cron job # - Fetch claimed items older than $n minutes/seconds # - Requeue or log&drop those timed-out items
Abstract interface class for a FIFO queue that keeps track of all in-flight ("claimed") items. Implementations are required to provide strict ordering and adhere to the run-time complexities listed below (or better).
The general workflow with
Queue::Q::ReliableFIFO based queue is:
To recover from failed consumers, one may apply in any one of many application specific recovery strategies such as periodically re-enqueuing all claimed items that are older than a threshold or possible simply clearing them out and logging the fact.
Since the actual recovery strategy is application-dependent and the support by the queue implementation may vary, there's no API for this in this abstract base class. That may change in a future release.
Given a data structure, that data structure is added to the queue. Items enqueued with
enqueue_item in order must be returned by
claim_item in the same order.
Given a number data structures, enqueues them in order. This is conceptually the same as calling
enqueue_item multiple times, but may help save on network round-trips.
Complexity: O(n) where n is the number of items to enqueue.
Returns the oldest item in the queue. Returns
undef if there is none left.
Implementations may (but should not) deviate from the strict O(1) complexity for the number of claimed items at any given time. That is acceptable as high as O(log(n)). This is likely acceptable because the number of items being worked on is likely not to be extremely large. Implementations that make use of this relaxed requirement must document that clearly. The number of queued items must not affect the complexity, however.
enqueue_items is to
claim_items is to
claim_item. Takes one optional parameter (defaults to 1): The number of items to fetch and return:
my @items = $q->claim_items(20); # returns a batch of 20 items
If there are less than the desired number of items to be claimed, returns a correspondingly shorter list.
Complexity: O(n) where n is the number of items claimed.
See the documentation of
claim_item for a potential relaxation on the complexity bounds with respect to the number of in-flight, claimed items.
Given a item that was previously claimed from this queue, it is removed from the claimed-items tracking and thus removed from the queue altogether.
Complexity: Generally O(1), but O(log(n)) under relaxed requirements (see above).
Given any number of items that were previously claimed from this queue, they are removed from the claimed-items tracking and thus removed from the queue altogether.
Complexity: Generally O(n), but O(n*log(n)) under relaxed requirements (see above) with n understood to be the number of items to mark as done.
Returns the number of items available in the queue.
Removes all content in the queue.
Complexity: O(n) where n is the number of items in the queue.
Herald van der Breggen, <firstname.lastname@example.org>
Copyright (C) 2012 by Herald van der Breggen
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.1 or, at your option, any later version of Perl 5 you may have available.