Thoughts about future releases, in no particular order.
Feel free to share your thoughts with me at mob@cpan.org
or through http://rt.cpan.org/NoAuth/Bugs.html?Dist=Forks-Super
Possible TODOs:
_x_ support sockets/pipes for cmd-style forks like IPC::Open3 does.
_X_ for Unixy systems
___ for MSWin32
IPC::Open3 supports MSWin32/os2 with the <system 1, @_>
construction. Don't think that is appropriate for this module.
___ There's enough stuff in here now that performance is affected,
especially when there are lots of short tasks. What can be done
to tighten up performance? What can be done to reduce overhead
when there are many short tasks?
___ disable $FSJ::Ipc::USE_TIE_?H? need to benchmark
_X_ tighten some delays in file-based IPC
___ shorten $Forks::Super::Queue::QUEUE_MONITOR_FREQ
___ fork { run => [ \@cmd, ... ] }
to invoke or emulate IPC::Run's rich feature set
___ What else can I learn by studying IPC::Run?
___ What other key-value pairs should FSJ::read_stdxxx handle?
_X_ warn => 0|1
_X_ timeout => max number of seconds to wait for input
___ log => 0|1|*handle|\@list?
___ No complaints yet, but is there a smarter way to
set the IPC directory?
___ Refactor how we create and remove the temp IPC directory.
_X_ Runtime IPC cleanup routine for long running programs. For long
completed jobs: close the open filehandles that have slipped
through the cracks; remove the IPC files; move from
%ALL_JOBS,@ALL_JOBS to %ARCHIVED_JOBS,@ARCHIVED_JOBS
___ Demos/examples/cookbook section.
___ Perform 1000's of jobs, 20 at a time
___ with queueing to perform other tasks
___ example: web crawler
_x_ example: multi-threaded du
_x_ timeout long running jobs
_x_ manipulate CPU affinities
___ dependencies
___ interactive client/server example of IPC
___ run a server using Forks::Super
_x_ see t/forked_harness.pl
_X_ load management
_X_ block while system is busy
_X_ suspend/resume
_X_ suspend/resume callback
___ bg_eval, bg_qx examples
_X_ factorial.pl for bg_eval
___ can_launch examples
___ how to: use sleep with Forks::Super
___ how to: use alarm with Forks::Super
___ changing IPC_DIR
___ tuning Forks::Super for fast jobs, slow jobs,
memory intensive jobs, cpu intensive jobs,
I/O bound jobs
___ scheduler app that can run for days at a time
___ reuse
___ share
___ daemon
___ POSIX::RT::Timer as possible replacement for get/set itimer?
___ Does anything bad happen when you set $SIG{CHLD} = 'IGNORE' ?
'DEFAULT'? sub {} ? undef ?
_X_ Yes. ANY setting for $SIG{CHLD} will let system calls
like sleep get interrupted by SIGCHLD events. Maybe.
See t/31.
It's curious that we can assign to $XSIG{CHLD}[-1]
(which does set a handler for SIGCHLD) but that DOESN'T
trigger interruption of system calls. It's as if perl
is just checking defined $SIG{CHLD} instead of whether
an actual signal handler is registered to decide whether
to interrupt sleep.
___ Setting $SIG{CHLD}=undef makes FS subtly wrong, though
setting $SIG{CHLD}=\&bogus is ok. How to keep $SIG{CHLD}
from getting undefined?
` ___ Override $Signals::XSIG::SIGTie::STORE, ::DELETE ?
___ Should we periodically set $SIG{CHLD}=\&garbage?
___ Yes again. When $SIG{CHLD}='IGNORE', calling wait or waitpid
NEVER returns a pid; it's always either 0 or -1. See
t/drop-in-exercise.pl (actually, this is platform dependent).
___ Should Forks::Super check the value of $SIG{CHLD}
and emulate this behavior?
___ Actually, this would be part of a good XSIG framework workout.
Some scripts with natural forks and wait/waitpid calls
should produce the same results with and without Forks::Super
(i.e., F::S is a drop-in replacement)
_X_ Test F::S as a drop in replacement to a program that
has a SIGCHLD handler. -- looks good
___ INET sockets as well as UNIX sockets, if you can commit to a port
before fork'ing and not bind to it until after fork'ing. Or if you
can pass the port from parent to child with a pipe?
___ reinstate t/31? ok on Cygwin, MSWin32, Linux, FreeBSD ?
___ still doesn't work on Linux?
___ a "wrapper" script that executes an arbitrary command in a
separate process, but as if it had come from a fork call within
a program using Forks::Super. The purpose is to make sure that
a program runs in a detached process. The wrapper will set up
all the (almost surely file based) IPC and then run the desired
program. The other purpose is to execute a command on a remote
host.
___ share MAX_PROC across processes, or have a SHARED_MAX_PROC
attribute to limit the number of processes across a group
of parent processes using Forks::Super.
___ defer to 1.0?
___ and the dual of this problem -- manage multiple
processes on separate remote hosts from a single
process (i.e., use Forks::Super to manage a grid?)
_O_ are %CHILD_STDxxx variables obsolete? How to deprecate?
Not obsolete. At least not until we remove the setting
$Forks::Super::OVERLOAD_ENABLED.
_o_ setuid => option to fork
_o_ Make forked_harness.pl an application that gets installed
with Forks::Super
_o_ or maybe not, people already use prove
_X_ put pod in t/forked_harness.pl
___ daemon support depends on file-based IPC?
make a socket-based alternative. Let $job->{signal_ipc}, {daemon_ipc}
be sockets?
_X_ FSJ::OS::Win32::signal_procs: is process group applied inconsistently?
___ Since MSWin32 doesn't have proper process groups, does it matter?
Yeah, it does. We should try and DWIM w.r.t. process groups/Win32.
___ Anything to learn from python multiprocessing module? See
stackoverflow.com/questions/7931455/
_X_ synchronization objects, acquire and release methods
___ perm fix for 48b with sync?
___ anything else
___ option channel => $nchannel
set up $nchannel bi-directional IPC channels with a background process.
In parent, $job->write_channel($k,$message) and
$msg=$job->read_channel($k).
In child, write_channel($k,$msg) and $msg=read_channel($k)
For low volumes, use pipes and sockets.
For high volumes, make channels use files.
___ exercise every method of FS::Tie::IPCDupSTDIN
(__config_fh_child_stdin_file, $job->{fh_config}{f_in},
sub/natural-style)
___ use shared memory between processes, where supported
___ synchronization based on shared memory
___ process pools based on shared memory
___ encryption and other layers on IO channels
___ parent_dump enhancements:
_X_ get and display stack trace of natural/sub-style children
___ measure input and output for IPCxxxHandle classes
___ CPAN testers find lots of timing errors in openbsd. Is pause(n)
on openbsd prone to returning significantly more or less than n
seconds later? Would a busy wait just for openbsd make things
better or worse?
_X_ Emulation mode, good for debugging, where calling fork()
does not actually create a new subprocess, but just runs
the child code in the main process.
_X_ RT#78285 - monitor open filehandles. When we approach the
limit, we may want to close the filehandles from old, finished
jobs.
_X_ $ON_TOO_MANY_OPEN_FILEHANDLES -- needs test
___ async method like threads, forks, Coro?
___ $job->{child_stdin}, {child_stdout}, {child_xxx} should be
restructured as $job->{child_fh}{stdin}, {stdout}, etc.
i.e., a dedicated member that holds a collection of iohandles
_X_ New syntax:
PREFORK BLOCK
POSTFORK BLOCK
POSTFORK_PARENT BLOCK
POSTFORK_CHILD BLOCK
Specifies code to be executed before and after fork calls.
What can this be used for?
* closing/opening filehandles
* closing/opening database connections
* closing/opening anything that doesn't play nice across
a fork
___ what happens if you mix Forks::Super and forks?
I'm guessing it's not good.
meh, it matters less than I thought
t/forked_harness.pl times out after 30 seconds but tests pass
regular make test worked?
t/24e-kill.t does not terminate
___ have a "remote_host" option to go with "cmd" option
_X_ remote_host => host
_X_ remote_host => [host, user, pwd]
to run on a remote server that requires some credentials
_X_ remote_host => [ \@hosts ] or [ \@hosts, user, pwd ]
to run on any of a cluster of available remote hosts
_X_ allow a separate MAX_PROC-like setting to be used for
remote hosts
___ remote_host_proto => 'ssh'
default is ssh and requires Net::OpenSSH
___ FS::Config::enable_signal_config not being tested?
_X_ Improve global destruction like I did in Devel::DumpTrace
___ Pool, Queue objects to share data between parent and children (workers)