PTools::Proc::NWay - Run a list of tasks with concurrent processing
This document describes version 0.08, released October, 2005.
Create a small script to use this module. Then run the small script using the '-h' (--help) command line option for usage help. The 'nway' command is just such an implementation. Exempli gratia:
use PTools::Local; # PerlTools Local module use PTools::Proc::NWay; # include this class exit( run PTools::Proc::NWay ); # return status to OS
nway [<options>] { -K | -R } filename [ - command [ -arg [...]]] where <options> include -c <num> - concurrent sessions (*) default is 8, max is 40 -L <str> - specify logfile prefix default is to use 'CMD_' -l <dir> - log dir for the output default is /tmp/nway.<PID> -m <secs> - max seconds for a task default is unlimited time -p [<num>] - preview of commands (+) default is to run commands -r - randomize the task list default is original order -h - display usage help text and ignore other options -F <char> - IFS char for data file default is a colon (':') -D [<num>] - enable debugging output default is no debug output where { -K | -R } - specifes whether to keep (-K) or remove (-R) the temporary output logfiles created for each task filename - specifies a file of tasks to perform concurrently use a value of "stdin" to read from STDIN file [ - command ] - is a command used to run each of the task lines [ -arg [...]] - is a list of opts/args passed to the <command> Note: (*) Default and maximum values for '-c <num>' may be calculated for each server based on the number of CPUs. These limits can vary depending on the class of the host machine. However, the actual maximum value used will be the lesser of the following: A) calculated maximum, B) tasks to run, or C) value for 'num' Note: (+) The '-p' (--preview) option can be used as '-p <num>' where 'num' is a number equal to 2 or more. When used in this manner, will cause a random (1 .. num) second delay in the preview to simulate tasks running with various ending times.
See the Options and Arguments section, below, for notes on using the long form of these options. For example, the following two commands are equivalent.
nway -K -l /tmp/xyzzy -L ECHO_ filename nway --logdir /tmp/xyzzy --LogPrefix ECHO_ --Keep filename
This module implements a limited concurrency mechanism whereby a list of commands can be run in sequence such that only a few of the commands are allowed to run concurrently. This provides a configurable throttle to prevent many simultaneous commands from overwhelming a system while still allowing a controlled amount of parallel processing.
This document assumes that a script named 'nway' is used to invoke this module. Much of the discussion and examples herein refer to an 'nway' script.
This concurrency mechanism reqires an input file of tasks to run. There is a lot of flexibility in how this file is created and used depending on the Arguments and Options specified. A complete description is included of both the Simple Case and of ways to use Named Input Fields in the data file.
Logging of output can be a bit confusing. There is the output from the main Driver module (sent to the 'nway' script's STDOUT), and the output from each command that is run by the Driver. Read the Output Logging section, below, for the full details.
When the list of tasks is completed a summary is generated. This includes a list of any tasks that failed for any reason. The summary also includes a count of the number of tasks, the start time and end time, the 'cumulative' time of all the tasks, and the 'elapsed' time of the 'nway' script. These last two can be used to determine the effectiveness of running tasks in parallel vs. running them each serially.
This is the only public interface to this class, and it does not accept any parameters. The Module Synopsis above shows a complete implementation of this class.
This class contains no public methods other than those described above.
The raison d'etre of 'nway' is to improve the throughput of a long list of tasks while limiting the overall impact to the system on which they are running.
However, given a long list of tasks, not every situation calls for an 'nway' solution. Tasks that have a very short duration could better be run sequentially, unless the logging and error detection features provided by 'nway' outweigh the extra overhead incurred.
Take 400 'echo xyzzy' commands. Running them sequentially using Perl's 'system' function can complete in a total of 2 seconds (for example), when output is STDOUT. Running these through 'nway' on the same machine gives comparatively dismal results. Cumulative times can range between 12 and 35 seconds, while elapsed times can be between 13 and 15 seconds. Using various concurrency levels does not have much effect. (Times listed are provided as an example only. Actual times can vary greatly, depending on the system used and the current system load.)
Bottom line: This module is not a panacea. It is best to do a little bit of experimenting before assuming that this module will speed up the overall completion time for a given list of tasks.
A number of command line options are available to control the throttling mechanism and other behaviors of this module. When using the abbreviated versions, options can be bundled using POSIX complient syntax. Options may not be permuted. Each of the following examples are equivalent. Again, this assumes that this module is implemented via a script named 'nway.'
nway -K -rm600D2 input.dat nway -K -m 600 -r --Debug 2 input.dat nway -Keep --randomize --maxtime 600 -D 2 input.dat
One of -K (--Keep) or -R (--Remove) and the filename argument are required while all others are optional. Note that using additional arguments can alter the nature of how the filename is parsed and used.
You must select one of -K (--Keep) or -R (--Remove) but not both. When using the -p (--preview) option, these arguments are optional.
This required argument specifies whether to keep (-K) or remove (-R) the temporary log files created for each task that is run. See the -l dir (--logidr dir) option, below, for notes on specifiyng the log directory in which to place the temporary log files. This argument will effect the the resulting output from the nway script.
It is also possible to use output redirection for each task separately. See the Output Logging section, below for details.
This required argument specifies a file of tasks to perform concurrently. Use a string value of 'stdin' for data to be read from STDIN. See the Configuration Data section for possible formats for this input file.
This optional argument specifies a command to run. When used, the command name must be preceeded with a dash ('-'). This allows any arguments to be added to the nway command line, even when they would not be valid options to this module.
Note that, when this argument is used, the contents of the data file are interpreted slightly differently. See the Configuration Data section, below for details. Also see the Custom Command Processing section for a variation on this usage.
When this optional argument list is used it may contain any arbitrary options and/or arguments that are valid for the specified command. This list is passed to the command before any task list options and/or arguments are added.
For each option, the long and short names are equivalent, but only one should be used at a time.
Specify the number of concurrent sessions. The default and maximum values will vary depending on the current host. Use the '-h' (--help) option to see the range on a given machine.
The default and maximum values for 'num' may be calculated for each server based on the number of CPUs. These limits can vary depending on the class of the host machine. However, the actual maximum value used will be the lesser of the following:
A) calculated maximum, B) tasks to run, or C) value for 'num'
Use this option to specify an alternate logfile prefix. The default logfile prefix is 'CMD_' for each file. In either case, the logfile suffix will be 'nnnnnn', a six-digit number that is the 'nway' internal number for each given task. See the Output Logging section, below for details.
Specify the log dir used to collect the output for each of the 'sessions' or 'tasks' executed by this module. By default, each session's command output is redirected into the /tmp/nway.<PID> directory, where <PID> is the process identifier of the parent nway session.
When this option is used and the named dir leaf does not exist, this module will created it prior to running the first task. Upon completion of the final task, if the -R (--Remove) option was also used, this module will remove the named dir leaf.
If the named dir leaf already does exist, this module will not remove the named dir upon completion. The log directory leaf will only be removed by this module when it is created by this module. See the Output Logging section, below for further details.
Specify the maximum seconds that a single task is allowed to run. The default is unlimited time. This global setting can be overridden on a task by task basis via the maxtime field name, shown below.
Tasks that exceed the specified number of seconds will be killed and a 'Timeout' error will be generated as the result for those tasks.
Preview the command list without actually running the commands. When previewing tasks, the -K (--Keep) and -R (--Remove) arguments are optional. By default, when the optional [num] is omitted, all commands to be run are previewed.
This option can be used as '-p <num>' where 'num' is a number equal to 1. When used in this manner, will cause the first task in the list to be 'previewd' and then the script will exit. This is useful to ensure that the various task components are assembled correctly.
This option can also be used as '-p <num>' where 'num' is a number equal to 2 or more. When used in this manner, will cause a random (1 .. 'num') second delay in the preview to simulate tasks running with various ending times. This mode is useful for testing the '-m secs' (--maxtime secs) option.
Randomize the task list. In some cases this can improve the throughput of the list of commands. Default is to run the commands in the original order.
An example of using this to imporove throughput could include a situation where sorting a list of tasks by user name could cause 'power users' tasks to clog all available concurrent sessions with long running tasks. In this case a random sort would allow shorter running tasks to intermix with longer running tasks, allowing more tasks to complete in a given time period.
Display usage help text and ignore other options.
This option is only meaningful when used with a 'field separated' input file and 'named fields' as described in the Configuration Data section. The default field separator character, when named fields are used, is the colon (':') character.
Note that a 'white space' character, such as a space or tab, may be used as a delimiter. In this case, take good care to ensure that only one of the characters appears between each field in the input.
White space characters can be used singly, as a 'space', a 'tab', or Perl's special '\s' meta character. In addition multiple white space characters can be specified using Perl's special '\s+' syntax. When entering these on the command line, make sure to either quote the string or add an extra escape character, such as \\s or \\s+ to ensure results are as intended, but don't do both. A warning will be printed if an invalid IFS character is detected.
-F " " -F " " # single space or tab character -F \\s -F "\s" # single space or tab character -F \\s+ -F '\s+' # one or more space and/or tab chars
No matter what character is used to delimit fields within a record always use a colon character to delimit field names in the '#FieldNames' header within a data file.
Enable debugging output. When 'num' is used, an increasing amount of output is displayed. A value of 2 or 3 is generally sufficient. The default is no debug output.
This class requires input from a data file (or STDIN) that is expected to be a list of tasks (commands) for this class to run. The tasks that are run get assembled from various components, depending on how a script using this class is invoked, and whether or not the data file contains named fields.
In the simple case, where named fields are not used within the input file, each entire line is used as one 'command string and argument list'.
However, when the [ - command [ -arg [...]]] form of this script is used, in the simple case, each line will be used as an 'argument list'. This list is passed to the named command after any options and/or arguments that are entered on the command line.
This class also allows for named input fields in the data file. If named fields are used, they must be used in the manner described in this section. When output from a command will be lengthy redirection can, and probably should, be included in the argument list. This is true either when using the simple case or when using named fields. See the Output Logging section for details.
On a line prior to the first data record in the input file, a line in the following format must be added to name each field. Optionally, an IFS character can be defined as well. The IFS character can be any non-numeric character, including 'white space' characters such as space and tab. When using white space characters, you probably want to add quotes, as shown in this example. When imbedded within a file, this will override any -F char (--IFS char) command line option value.
#FieldNames field1:field2:field3... #IFSChar " "
The '#FieldNames' tag must start in column one and a space must appear between this tag and the list of field names. The field names may not contain spaces and must always be separated by a colon (':') character, as shown in the line above, no matter what character is used to delimit fields in the data records. See the -F char (--IFS char) command line option, above. Field names can appear in any order but, obviously, the order in the 'FieldNames' list must match the field order in each data record.
Field names can include one or more of the following. They can be in 'lower' case or 'UPPER' case, but not 'Mixed' case. When using named fields, the only required field is 'taskcmd' (or 'TASKCMD'). Note that if a field is left empty for a given data record global command line settings may apply.
This class also allows for special cases with the '#IFSChar' header. White space characters can be used singly, as a 'space', a 'tab', or Perl's special '\s' meta character. In addition multiple white space characters can be specified using Perl's special '\s+' syntax. The IFS character can be quoted using double quotes within the header.
#IFSChar " " # single tab character #IFSChar " " # single space character #IFSChar "\s" # single space or tab character #IFSChar "\s+" # one or more space and/or tab characters
This implies that the double quote character can not be used as a field separator within a data file. Any other 'comment' lines, where the first non-blank character is a '#' character, are ignored, and any empty lines are ignored when reading the task list.
When specifying a single white space character, make sure that there is only one of them in between each field within a record.
When using named fields, taskcmd contains the command to be run and, optionally, any or all command options and arguments.
If used without other field names, this is equivalent to not using named fields as the entire line will still be used as one 'command string and argument list'.
When using named fields, taskcmd is the only required field unless the [ - command [ -arg [...]]] form of this script is used.
However, when the [ - command [ -arg [...]]] form of this script is used, in the case of named fields, each argN field, described next, is appended to any 'argument list' entered on the command line. In this case, any taskcmd field named in the task list is ignored.
Output from the command can be redirected using standard Bourne shell redirection syntax in either a taskcmd string or, when 'argument' fields are used, the final argN argument, discussed next. See the Output Logging section, below, before using output redirection.
Optionally, command options and arguments can be separated into one or more unique fields. Each field will be a separate element of the array passed to the system 'exec' when running a particular command. Any number of argument fields can be included but any gap in the numeric sequence will terminate the list.
Valid Example:
#FieldNames taskcmd:arg0:arg1:arg2
Invalid Example:
#FieldNames taskcmd:arg0:arg2:arg3
In this second example arg2 and arg3 fields will be ignored as there is no arg1 found in the '#FieldNames' list.
This optional field can be used to limit the total number of seconds that a particular command is allowed to run. This field will override the '-m secs' (--maxtime secs) command line option which can be used as a global time limit for commands that do not have a 'maxtime' entry in the data record.
A value of '0' (zero) indicates unlimited time, even when the -m secs (--maxtime secs) option is used, while a 'null' (empty) value will use the value specified by the -m secs option, if it was used, or unlimited time if the -m secs option was not used.
This optional field can be used to specify both the real and effective User ID with which to execute the command. For this to work, this module must be run with sufficient permissions to execute the setuid command.
Also see the statpath field, below.
This optional field can be used to specify both the real and effective Group ID with which to execute the command. For this to work, this module must be run with sufficient permissions to execute the setgid command.
Also see the statpath field, next.
This optional field can be used to specify the path to a data file from which the effective User ID and Group ID will be determined and then used to execute the taskcmd. For this to work, this module must be invoked with sufficient permissions to execute the setuid and setgid commands.
In addition, this field can refer to any other named 'argN' input field by using the symbolic construct '%argN%', where 'N' corresponds to the argument number. This brief example assumes that the command to run was named via a command-line argument.
#FieldNames arg0:statpath #IFSChar "\s+" /ClearCase/newview/markd %arg0% /ClearCase/newview/robinson %arg0%
Also see the uid and gid fields, above. Note that, when used, this field superceedes the other two fields.
As mentioned above, logging of the output generated by this module can be confusing. There is the output from the main Driver module (sent to the 'nway' script's STDOUT), and the output from each command that is run by the Driver.
The various options and inputs that effect output logging include the -l dir (--logidr dir) option, the -L string (--LogPrefix) option, the -K (--Keep) option, the -R (--Remove) option, output redirection characters included as arguments in the task list and, of course, any change to STDOUT and/or STDERR within a given task command itself.
By default, when none of the above are used, output of a command run for a given task list entry is redirected to a file named as follows.
/tmp/nway.<PID>/CMD_nnnnnn where <PID> is the nway process ID and nnnnnn is an internal sequence number of the command being run
If the default directory already exists, the 'nway' script will exit with an error. This module expects to create a new subdirectory 'leaf,' and considers it inappropriate to overwrite any prior log files. When the -l dir option is used, however, the module will assume that 'User Knows Best,' and will overwrite any or all files contained therein.
Using the various options listed above, the format and location of the file used for each task's output can be altered. For example,
nway --logdir /tmp/xyzzy --LogPrefix ECHO_ --Keep filename
will change the above default values such that output for the command run for a given task list entry is redirected to a file named as follows.
/tmp/xyzzy/ECHO_nnnnnn
You must select one of -K (--Keep) or -R (--Remove) but not both [unless using the -p (--preview) option--since no logging is done during preview, these arguments are optional in this case].
When the -K (--Keep) option is used, the temporary log files are kept in the logdir after each task completes. When the 'nway' script completes, the log directory is kept intact. Since the output from each session is kept, as each task completes only a note that includes the task's logfile name is included in the 'nway' output.
When the -R (--Remove) option is used, the temporary log files are removed from the logdir as each task completes. When the 'nway' script completes, the log directory is also removed. Since the output from each session is removed, as each task completes the entire contents of each task's logfile is included in the 'nway' output.
One additional note here. When using the -l dir (--logdir dir) option and the named dir already exists, this module will not remove the named dir upon completion. The log directory leaf will only be removed by this module when it is created by this module.
It is also possible to use standard Bourne shell redirection syntax within each command string or argument list to redirect each command's output as desired. When this is done, the resulting output is not copied to the 'nway' script's STDOUT. If you go to all the trouble of specifically redirecting output for each and every task to a separate log file, the assumption is that each log will contain a large amount of data. In this case only the file name is added to the 'nway' script's output upon completion of each task as a reminder of where to look for the logged data.
Of course, it is always possible for a command to redirect its own output during execution. In this case this module will not detect the output redirection and simply report that no output was found for the command.
In any and all cases, if a command fails to complete successfully, an error message is added to the 'nway' script's STDOUT. When output redirection is used it is then necessary to examine the specific log file to determine the exact error that caused the failure.
Note: This functionality may change in a future release.
The 'nway' throttling mechanism runs each task in a given 'slot' or queue entry location. Obtaining the number of the particular 'nway' queue entry from within a given task is occasionally useful. In this case there are two ways to obtain the slot number, which is refered to as an 'NRID.' As the example just below shows, NRIDs are zero-based numbers.
The first way to obtain the NRID is to use the symbolic construct '%nrid%', either in lower or upper case, anywhere in the arguments for a given command. Prior to execution of each command, the appropriate value will be substituted. This can be on the command line or in the input file whether using named fields or not.
The second way to obtain the NRID is within any script or command that is run by the 'nway' script. The environment variable 'NWAY_RESOURCE_ID' will be set to the appropriate value.
Note that the format of the NRID may vary slightly, depending on the concurrency level used. The value will be prepended with zeros to ensure that a consistent number of digits will be used throughout. This might become useful if a sort of the results, based on the NRIDs, is desired.
For example, when using a concurrency level of 4 ('-c 4') the NRIDs will be in the range '0 .. 3'. When using a concurrency level of 18 ('-c 18') NRIDs will be in the range '00 .. 17'.
This module recognizes the signals HUP, INT, and TERM. Currently, each of these signals will cause the Driver process to flush any remaining (not yet run) tasks from the list and then wait for all running tasks to complete. In addition, the INT signal is propagated to all running tasks, which will cause the tasks to exit.
It is also possible to run the 'nway' script that uses this Module in the following manner.
nway [<options>] [{ -K | -R }] Perl::Module [ - command [ -arg [...]]]
This allows a custom Perl::Module to replace the Proc::NWay::Session module that processes each task/command. The Perl::Module can appear anywhere that a normal command can be specified. This includes the task data input file in either the Simple Case or when using Named Input Fields in the data file. When used in this manner, the -K and -R arguments are optional.
When using this syntax, the Perl::Module must have at least one set of double colons ('::'). This is the mechanism that causes the Driver module to alter the normal command processing and load the named module. Be aware that custom module(s) is(are) loaded at run time so any compile errors will cause the Driver module to terminate in an error state. Currently when this happens, any running tasks will be left to fend for themselves.
For easiest results, the custom module can subclass Proc::NWay::Session and then override any or all of the methods and behaviors in the parent class. Unfortunately, a good example of this is not included yet.
Note well that using the custom command processing features voids any and all warranties for this product.
Not every situation calls for an 'nway' solution. Tasks that have a very short duration could better be run sequentially, unless the logging and error detection features provided by 'nway' outweigh the overhead incurred. See the section Negative Effects of Concurrency Processing, above.
Be sure to read the Output Logging section, above before using this module. Note that the logging functionality may change in a future release.
Since a 'white space' character, such as space(s) and/or tab(s), may be used as delimiters in the input file, take good care to ensure that only one of the characters appears between each field in the input, unless you are using the special '\s+' IFS syntax that allows multiple white space characters. And, in this case, make sure that no white space characters appear within a 'field.'
However, no matter what character is used to delimit fields within a record, always use a colon character to delimit field names in the '#FieldNames' header within the input file.
The output logfiles for each task are created starting with a six-digit number. If the task list exceeds this, additional digits are added as needed, and no output logfiles will be overwritten. However, this will cause the Unix 'ls' command to sort the log filenames incorrectly, but only when there are one million or more output logfiles in the directory.
And finally, using Custom Command Processing features voids any and all warranties for this product, either express or implied.
Diagram of the 'NWay' Task Throttling Architecture.
Parent proc runs the show session procs command procs ========================== ============== ============= ____________ ____________ ____________ _____________ | | | | | | | | | NWay |--| Driver |-----| Session |-----| task 1... | |__________| |__________|\ |__________| |___________| ^ \ ____________ _____________ _____^______ \| | | | | DataFile | | Session |-----| ...task n | |__________| |__________| |___________|
This class depends upon the set of POE classes. This class also depends on the PerlTools Architecture, a light framework to facilitate the creation of Perl tools. In addition an optional external command named numcpus can be used to calculate the default and maximum number of concurrent processes that will run.
The nway script is an implementation of this module as shown in the Module Synopsis section, above.
The following modules used by this class directly control the session processes (no man pages exist for these). See the modules PTools::Proc::NWay::Driver, PTools::Proc::NWay::Logger, PTools::Proc::NWay::Session and PTools::Proc::NWay::SessionLimits.
The following Perl Object Environment (POE) classes are used. See POE::Kernel, POE::Filter::Reference, POE::Session and POE::Wheel::Run.
In addition, the following PerlTools utility classes are used. See PTools::Counter, PTools::Date::Format, PTools::Debug, PTools::Options, PTools::Proc::Backtick, PTools::SDF::ARRAY, PTools::SDF::File, PTools::SDF::SDF, PTools::SDF::Lock::Advisory, PTools::SDF::Sort::Random, PTools::String and PTools::Time::Elapsed.
See also PTools::Local and PTools::Global for information about using the PerlTools framework.
Chris Cobb, <nospamplease@ccobb.net>
Copyright (c) 2005-2007 by Chris Cobb. All rights reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install PTools::Proc::NWay, copy and paste the appropriate command in to your terminal.
cpanm
cpanm PTools::Proc::NWay
CPAN shell
perl -MCPAN -e shell install PTools::Proc::NWay
For more information on module installation, please visit the detailed CPAN module installation guide.