The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Helios-2.80
River stage one • 5 direct dependents • 5 total dependents
2 ++

Changes for version 2.80 - 2014-03-15

  • New Features
    • New Jobtype System: Before Helios 2.80, the jobtype of a job had to be the same name as the service class that ran the job. If you had a service class you wanted to run multiple jobtypes, name a jobtype something different than the service class, or divide a workload in a complex way, you had to create multiple service classes by creating empty subclasses of your service class. This could lead to a lot of boilerplate code to generate and manage. With the new Helios::JobType class and helios_jobtype_add command, new jobtypes can be created as necessary for your jobs, and the new --jobtypes option of helios.pl allows your service to work with whatever jobtype(s) you need, regardless of the service's name. This also eliminates the need for the HELIOS_CLASS_MAP table, a virtually undocumented feature only used by the submitJob.pl CGI. This separation of services and jobtypes will also allow more advanced job queuing features in the future.
    • Worker Launch Patterns: Previous versions of Helios launched workers to service jobs in a one-at-time manner. With Helios 2.80, the algorithm used to determine the number of workers to launch is configurable, with 3 algorithms to choose from: "linear", "dynamic", and "optimistic". "Linear" is the old, one-at-time algorithm. The "dynamic" pattern attempts to keep as many workers running as there are available jobs in the queue, adjusting the number of workers launched based on available job count. The "optimistic" pattern launches as many workers as there are available jobs in the queue, regardless of the number of workers already running (limited by the MAX_WORKERS limit). The default is "linear" for backward compatibility reasons. Most Helios services should work well with the "dynamic" pattern, and it has been extensively tested in production environments. The "optimistic" pattern, while theoretically spooling up workers the fastest, is best used with services with large numbers of jobs of short duration (1-2 sec or less), as shared resources could be taxed and job contention may result. See the Helios::Configuration POD for more information.
    • New Commands: New shell commands greatly reduce the need to consult the Helios::Panoptes web interface or touch the Helios database directly with SQL commands: helios_config_get/set/unset helios_job_status helios_job_info helios_service_status Additionally, the new helios_config_import command allows you to bulk-import configuration parameters from an INI-style config file, easing deployment of services with lots of config params.
    • Helios::TS: Helios no longer relies directly on TheSchwartz job queuing system, but on its own Helios::TS superset system. This allows new Helios-specific features to be added (including some of the changes listed above).
    • Helios::Service->JobLockInterval(): This new method allows your service to specify the amount of time jobs are locked in the job queue. This was previously set at 1 hour unless you reached into the TheSchwartz layer and redefined the grab_for() setting, which caused problems if you wanted to use MaxRetries() and RetryInterval() to retry failed problems more frequently than once an hour. If you wish to retry failed jobs more frequently than once an hour, define a JobLockInterval() method in your service class with the same value as your RetryInterval(). ALWAYS REMEMBER to make sure these intervals always allow enough time to complete or fail a job; otherwise, two different workers can end up running the same job simultaneously.
    • Helios::Config: New get/set/unsetParam() methods retrieve individual config parameter values for a service, instead of having to retrieve the entire configuration hash. These are primarily used in the helios_config_* shell commands, but can be used by your own programs and scripts to access Helios config parameters outside of Helios services.
    • Job Prioritization: Normally with the Helios::TS job queuing system, workers pull jobs from the job queue at random to reduce job contention between workers. Using the PRIORITIZE_JOBS config parameter and Helios::Job->setPriority() method, you can set priorities for jobs. Due to the way Helios::TS works, this will not be a strict "one-after-the-other" priority, but still can be used in a general sense to give a job or group of jobs a higher chance of being run before another group of jobs.
  • Other Improvements
    • Helios::Config: Improvements have reduced the number of database queries by 50%. Additionally, a new get/setService() accessor method has been added for consistency; the old get/setServiceName() methods are deprecated but retained for backward compatibility.
    • Helios::Job: The Helios::Job API also receives some cleanup with the addition of 3 new accessor pairs: get/setArgString() - replaces get/setArgXML() get/setJobType() - replaces get/setFuncname() get/setJobtypeid() - replaces get/setFuncid() These new methods keep Helios::Job consistent with the rest of the Helios system and also allow for future changes. As with the Helios::Config changes, the old methods are deprecated but retained for backward compatibility.
    • helios_job_submit.pl: Added the --verbose option to echo the jobid of the job if job submission was successful. Echoing the jobid by default could cause backward compatibility problems, so it was decided to add an option instead. This behavior will most likely become the default in Helios 3.0.
    • Bugfix for [RT79690] - A patch to prevent this bug from causing lost jobs was incorporated into Helios 2.61, but a full fix required a change at the job queuing layer (and thus required Helios::TS). As mentioned previously in the Helios 2.61 Changes and the bug's RT thread, if you have experienced this bug you should also upgrade your DBD:: driver module as well.

Documentation

Helios
submitJob.pl
CGI script to receive jobs for Helios via HTTP POST
helios.pl
Launch a daemon to service jobs in the Helios job processing system
helios_config_get
get a config parameter's value from the Helios collective database
helios_config_import
import a INI-style config file into the Helios database
helios_config_set
set a config param's value in the Helios database
helios_config_unset
delete a config param's value from the Helios database
helios_job_info
get detailed information about a job in a Helios collective
helios_job_status
get a Helios job's exitstatus
helios_job_submit.pl
Submit a job to the Helios job processing system from the cmd line
helios_jobtype_add
Add a jobtype to the Helios collective database
helios_log_clean.pl
Clean old log and history entries from the Helios database
helios_service_status
check the status of services running in a Helios collective
Helios::Configuration
Helios configuration parameter reference
Helios::Tutorial
a tutorial for getting started with Helios

Modules

MP3IndexerService
Stub::StubService
Bundle::Helios
install all Helios related modules
Helios
a distributed job processing system
Helios::Config
base class for Helios configuration system
Helios::Error
a convenience class to import all Helios::Error exception classes
Helios::Error::BaseError
base exception class for Helios services
Helios::Error::ConfigError
exception class for Helios indicating a configuration error occurred
Helios::Error::DatabaseError
exception class for Helios indicating a database error occurred
Helios::Error::Fatal
fatal exception class for Helios indicating a job failed but can be re-attempted
Helios::Error::FatalNoRetry
fatal exception class for Helios indicating a job failed and the error was so serious the job should not be reattempted.
Helios::Error::InvalidArg
exception class for Helios indicating a job's args are invalid
Helios::Error::JobTypeError
exception class for Helios indicating a jobtype error occurred
Helios::Error::LoggingError
exception class for Helios indicating an error occurred in the logging subsystem
Helios::Error::ObjectDriverError
exception class for Helios indicating an object driver error occurred
Helios::Error::Warning
exception class for Helios indicating a job was successful but it encountered errors during processing
Helios::Job
base class for jobs in the Helios job processing system
Helios::JobType
class to represent Helios jobtypes
Helios::Logger
Base class for sending Helios logging information to external loggers
Helios::Logger::Internal
Helios::Logger subclass implementing Helios internal logging
Helios::MetajobBurstService
base class for metajob burst services in Helios
Helios::ObjectDriver
class to return Helios::ObjectDriver::DBI objects pointing to the collective database
Helios::ObjectDriver::DBI
Data::ObjectDriver subclass for Helios
Helios::Service
base class for services in the Helios job processing system
Helios::TS
TheSchwartz subclass for Helios
Helios::TS::Job
TheSchwartz::Job subclass for Helios
Helios::TestService
Helios::Service subclass useful for testing

Provides

Helios::ConfigParam
in lib/Helios/ConfigParam.pm
Helios::JobHistory
in lib/Helios/JobHistory.pm
Helios::LogEntry
in lib/Helios/LogEntry.pm
Helios::LogEntry::Levels
in lib/Helios/LogEntry/Levels.pm

Examples

Other files