Thoughts about future releases, in no particular order.
Feel free to share your thoughts with me at mob@cpan.org
or through http://rt.cpan.org/NoAuth/Bugs.html?Dist=Forks-Super
Possible TODOs:
_x_ support sockets/pipes for cmd-style forks like IPC::Open3 does.
_X_ for Unixy systems
___ for MSWin32
IPC::Open3 supports MSWin32/os2 with the <system 1, @_>
construction. Don't think that is appropriate for this module.
___ There's enough stuff in here now that performance is affected,
especially when there are lots of short tasks. What can be done
to tighten up performance? What can be done to reduce overhead
when there are many short tasks?
___ disable $FSJ::Ipc::USE_TIE_?H? need to benchmark
_X_ tightened some delays in file-based IPC
___ shorten $Forks::Super::Queue::QUEUE_MONITOR_FREQ
___ fork { run => [ \@cmd, ... ] }
to invoke or emulate IPC::Run's rich feature set
___ What else can I learn by studying IPC::Run?
___ What other key-value pairs should FSJ::read_stdxxx handle?
_X_ warn => 0|1
_X_ timeout => max number of seconds to wait for input
___ log => 0|1|*handle|\@list?
___ No complaints yet, but is there a smarter way to
set the IPC directory?
___ Refactor how we create and remove the temp IPC directory.
___ Runtime IPC cleanup routine for long running programs. For long
completed jobs: close the open filehandles that have slipped
through the cracks; remove the IPC files; move from
%ALL_JOBS,@ALL_JOBS to %ARCHIVED_JOBS,@ARCHIVED_JOBS
___ Demos/examples/cookbook section.
___ Perform 1000's of jobs, 20 at a time
___ with queueing to perform other tasks
___ example: web crawler
_x_ example: multi-threaded du
_x_ timeout long running jobs
_x_ manipulate CPU affinities
___ dependencies
___ interactive client/server example of IPC
___ run a server using Forks::Super
_x_ see t/forked_harness.pl
_X_ load management
_X_ block while system is busy
_X_ suspend/resume
_X_ suspend/resume callback
___ bg_eval, bg_qx examples
_X_ factorial.pl for bg_eval
___ can_launch examples
___ how to: use sleep with Forks::Super
___ how to: use alarm with Forks::Super
___ changing IPC_DIR
___ tuning Forks::Super for fast jobs, slow jobs,
memory intensive jobs, cpu intensive jobs,
I/O bound jobs
___ scheduler app that can run for days at a time
___ reuse
___ share
___ daemon
_x_ Forks::Super::Job::dispose( @jobs ) method
_X_ Removes entry from @ALL_JOBS, %ALL_JOBS
___ move to @ARCHIVED_JOBS, %ARCHIVED_JOBS?
___ POSIX::RT::Timer as possible replacement for get/set itimer?
___ Does anything bad happen when you set $SIG{CHLD} = 'IGNORE' ?
'DEFAULT'? sub {} ? undef ?
_X_ Yes. ANY setting for $SIG{CHLD} will let system calls
like sleep get interrupted by SIGCHLD events. Maybe.
See t/31.
It's curious that we can assign to $XSIG{CHLD}[-1]
(which does set a handler for SIGCHLD) but that DOESN'T
trigger interruption of system calls. It's as if perl
is just checking defined $SIG{CHLD} instead of whether
an actual signal handler is registered to decide whether
to interrupt sleep.
___ Setting $SIG{CHLD}=undef makes FS subtly wrong, though
setting $SIG{CHLD}=\&bogus is ok. How to keep $SIG{CHLD}
from getting undefined?
` ___ Override $Signals::XSIG::SIGTie::STORE, ::DELETE ?
___ Should we periodically set $SIG{CHLD}=\&garbage?
___ Yes again. When $SIG{CHLD}='IGNORE', calling wait or waitpid
NEVER returns a pid; it's always either 0 or -1. See
t/drop-in-exercise.pl (actually, this is platform dependent).
___ Should Forks::Super check the value of $SIG{CHLD}
and emulate this behavior?
___ Actually, this would be part of a good XSIG framework workout.
Some scripts with natural forks and wait/waitpid calls
should produce the same results with and without Forks::Super
(i.e., F::S is a drop-in replacement)
_X_ Test F::S as a drop in replacement to a program that
has a SIGCHLD handler. -- looks good
___ INET sockets as well as UNIX sockets, if you can commit to a port
before fork'ing and not bind to it until after fork'ing. Or if you
can pass the port from parent to child with a pipe?
___ refactoring needed after getting daemon code to work.
_x_ handle failures
___ reinstate t/31? ok on Cygwin, MSWin32, Linux, FreeBSD ?
___ still doesn't work on Linux?
_O_ Test file just for Forks::Super::XXX
_X_ Forks::Super::Util
_X_ Forks::Super::Queue
_X_ Forks::Super::Job
_O_ Forks::Super::Wait nothing testable in isolation
except _cleanse_waitpid_arg
_x_ Currently, 'kill' to a fork-to-cmd or fork-to-exec might signal
two separate processes. Can we coerce kill to return 1 in this
case instead of 2? Currently it is a big incompatibility between
this module and core use.
_?_ just signal $job->signal_pid, not $job->{real_pid}?
I think that works. When signal_pid != {real_pid}, {real_pid}
is a pretty thin wrapper around signal_pid and will not
last long when signal_pid is terminated. Still, you want to
be wary of calling kill ...,$job and immediately checking
___ a "wrapper" script that executes an arbitrary command in a
separate process, but as if it had come from a fork call within
a program using Forks::Super. The purpose is to make sure that
a program runs in a detached process. The wrapper will set up
all the (almost surely file based) IPC and then run the desired
program. The other purpose is to execute a command on a remote
host.
___ share MAX_PROC across processes, or have a SHARED_MAX_PROC
attribute to limit the number of processes across a group
of parent processes using Forks::Super.
___ defer to 1.0?
___ and the dual of this problem -- manage multiple
processes on separate remote hosts from a single
process (i.e., use Forks::Super to manage a grid?)
_O_ are %CHILD_STDxxx variables obsolete? How to deprecate?
Not obsolete. At least not until we remove the setting
$Forks::Super::OVERLOAD_ENABLED.
_o_ setuid => option to fork
_o_ Make forked_harness.pl an application that gets installed
with Forks::Super
_o_ or maybe not, people already use prove
_X_ put pod in t/forked_harness.pl
___ daemon support depends on file-based IPC?
make a socket-based alternative. Let $job->{signal_ipc}, {daemon_ipc}
be sockets?
___ FSJ::OS::Win32::signal_procs: is process group applied inconsistently?
___ Since MSWin32 doesn't have proper process groups, does it matter?
Yeah, it does. We should try and DWIM w.r.t. process groups/Win32.
_X_ In Java, you can send SIGQUIT to a virtual machine and the JVM will
dump the stack trace for every thread with some other data about the
program. Can we do something similar for Forks::Super?
_X_ not enabled by default
_X_ install sighandler in every natural/sub child to write stack trace
___ How to test?
___ Anything to learn from python multiprocessing module? See
stackoverflow.com/questions/7931455/
_X_ synchronization objects, acquire and release methods
___ perm fix for 48b with sync?
___ anything else
___ option channel => $nchannel
set up $nchannel bi-directional IPC channels with a background process.
In parent, $job->write_channel($k,$message) and
$msg=$job->read_channel($k).
In child, write_channel($k,$msg) and $msg=read_channel($k)
For low volumes, use pipes and sockets.
For high volumes, make channels use files.
___ exercise every method of FS::Tie::IPCDupSTDIN
(__config_fh_child_stdin_file, $job->{fh_config}{f_in},
sub/natural-style)
___ use shared memory between processes, where supported
___ synchronization based on shared memory
___ process pools based on shared memory
___ encryption layers on IO channels
___ parent_dump enhancements:
_X_ get and display stack trace of natural/sub-style children
___ measure input and output for IPCxxxHandle classes
___ CPAN testers find lots of timing errors in openbsd. Is pause(n)
on openbsd prone to returning significantly more or less than n
seconds later? Would a busy wait just for openbsd make things
better or worse?
_x_ Signal and conf file based controls. For example,
sending SIG45 followed by SIG46 (within 500 ms) means
"increase $Forks::Super::MAX_PROC by 1",
SIG45+SIG45 means "reload config from file $ENV{FORKS_SUPER_CONFIG}".
possible asynchronous operations:
_x_ reload config file
_x_ increase $Forks::Super::MAX_PROC
_x_ decrease $Forks::Super::MAX_PROC
_x_ increase $Forks::Super::MAX_LOAD
_x_ decrease $Forks::Super::MAX_LOAD
_x_ dump (see Forks::Super::Debug::parent_dump)
___ what else?
___ Emulation mode, good for debugging, where calling fork()
does not actually create a new subprocess, but just runs
the child code in the main process.
___ RT#78285 - monitor open filehandles. When we approach the
limit, we may want to close the filehandles from old, finished
jobs.
___ in forked-harness.pl, have option to return tests in order
_X_ alternate ways of invoking fork to support?
_X_ fork sub { ... }
fork sub { ... }, %options
fork sub { ... }, \%options
fork \&code [, %options | \%options ]
These all run the specified Perl code in a bg proc, as if
$options->{sub} = BLOCK were specified
_X_ fork [ @cmd ]
fork [ @cmd ], %options | \%options
Fork and run a command in the background, as if
$options->{cmd} = \@cmd where specified
_X_ http://stackoverflow.com/questions/20415186
on_busy => 'queue' should be the default in many contexts.
In some contexts, it is the only sensible value, like
_X_ jobs with dependencies
_XX_ jobs with delay / start_after