Hopkins - complete multiqueue job scheduling and execution system
Hopkins is, in simplest terms, a better cron. In depth, though, Hopkins is an extensible application geared toward the management, scheduling, and execution of both inline perl as well as external binaries.
Hopkins's advantages include:
There are many job management systems out there, varying in complexity. Hopkins was designed to be simple to understand and simple to configure.
While Hopkins is written in perl and will dynamically load, instantiate, and execute any object that provides a "run" method, it does not require your class to be aware of its environment. In fact, Hopkins does not require your task to be written in perl at all.
Hopkins ships with Hopkins::Config::XML, which allows your queues and tasks to be defined entirely in XML, validated before load via XML Schema. However, Hopkins::Config may be subclassed to provide configuration any way you like.
Hopkins also provides a mechanism by which your config may be scanned for changes. For example, Hopkins::Config::XML periodically checks the configuration file for changes and validates the XML before replacing the existing config.
Hopkins supports an infinite number of queues that each have their own concurrency limits and behaviors upon task failure. Per-queue concurrencies allow you to define serial queues (concurrency=1) or worker queues (concurrency > 1).
Each queue's behavior on task failure is configurable. For example, queues may be configured to halt upon task failure, stopping the execution of queued tasks until a human has a chance to examine the failure.
Each configured task may have a number of schedules defined, including none.
Each queued task records state information to a configurable database backend. Information stored includes enqueue time, time to execute, execution time, completion time, status flags, as well as all output generated on stdout and stderr.
If the database backend ever becomes unavailable, Hopkins will continue to run, queueing up database requests.
Hopkins supports loadable plugins which may hook into the running system by making use of POE. Two plugins include a web-based user interface (Hopkins::Plugin::HMI) as well as a SOAP-based interface (Hopkins::Plugin::RPC).
Hopkins makes use of the excellent Log::Log4perl module for its logging, allowing you to direct task execution errors and other output to the appropriate parties with ease.
Queue (both task and database) contents are written to disk upon entering/exiting the queue. Thus, if the daemon were to be restarted for any reason, it will continue processing from where it left off.
Hopkins has been used in an environment handling > 60 tasks in four separate queues and does not eat resources. I make no promises as to its scalability beyond that.
See Hopkins::Manual::Configuration::XML for information on configuring Hopkins using the default XML configuration.
instantiates a new Hopkins object. the Hopkins constructor accepts a hash of options. currently supported options are:
path to the hopkins XML configuration file
path to the log4perl configuration file
configuration scan resolution (in seconds)
scheduler poll resolution (in seconds)
start the hopkins daemon. this method will never return.
returns a truth value indicating whether or not a session exists with the specified alias.
returns a list of currently active session aliases
DateTime::Format::ISO8601->parse_datetime wrapper that traps exceptions. this really shouldn't be necessary.
returns a Log::Log4perl logger for the current session. the get_logger expects the POE kernel to be passed to it. if no POE::Kernel is passed, it will default to $poe_kernel.
this is my first foray into POE territory. the way the system is architected may be horribly inefficient, cause cancer, or otherwise be a general nuisance to its intended user(s). my bad.
Mike Eldridge <email@example.com>
Copyright (c) 2010 Mike Eldridge. All rights reserved.
This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself.