NAME
Urlader - installer-less single-file independent executables
SYNOPSIS
use Urlader;
DESCRIPTION
Urlader (that's german for "bootloader" btw.) was created out of
frustration over PAR always being horribly slow, again not working,
again not being flexible enough for simple things such as software
upgrades, and again causing mysterious missing file issues on various
platforms.
That doesn't mean this module replaces PAR, in fact, you should stay
with PAR for many reasons right now, user-friendlyness is one of them.
However, if you want to make single-file distributions out of your perl
programs (or python, or C or whatever), and you are prepared to fiddle a
LOT, this module might provide a faster and more versatile deployment
technique then PAR. Well, if it ever gets finished.
Also, *nothing in this module is considered very stable yet*, and it's
far from feature-complete.
Having said all that, Urlader basically provides three services:
A simple archiver that packs a directory tree into a single file.
A small C program that works on windows and unix, which unpacks an
attached archive and runs a program (perl, python, whatever...).
A perl module support module (*this one*), that can be used to query the
runtime environment, find out where to install updates and so on.
EXAMPLE
How can it be used to provide single-file executables?
So simple, create a directory with everything that's needed, e.g.:
# find bintree
bintree/perl
bintree/libperl.so.5.10
bintree/run
bintree/pm/Guard.pm
bintree/pm/auto/Guard/Guard.so
bintree/pm/XSLoader.pm
bintree/pm/DynaLoader.pm
bintree/pm/Config.pm
bintree/pm/strict.pm
bintree/pm/vars.pm
bintree/pm/warnings.pm
# cat bintree/run
@INC = ("pm", "."); # "." works around buggy AutoLoader
use Guard;
guard { warn "hello, world!\n" }; # just to show off
exit 0; # tell the urlader that everything was fine
Then pack it:
# wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86
# urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \
LD_LIBRARY_PATH=. ./perl run \
>myprog
# chmod 755 myprog
The packing step takes a binary loader and appends all the files in the
directory tree, plus some meta information.
The resulting file is an executable that, when run, will unpack all the
files and run the embedded program.
CONCEPTS
urlader
A small (hopefully) and relatively portable (hopefully) binary that
is prepended to a pack file to make it executable.
You can build it yourself from sources (see prebuilt/Makefile in the
distribution) or use one of the precompiled ones at:
http://urlader.schmorp.de/prebuilt/1.0/
The README there has further information on the binaries provided.
exe_id
A string that uniquely identifies your program - all branches of it.
It must consist of the characters "A-Za-z0-9_-" only and should be a
valid directory name on all systems you want to deploy on.
exe_ver
A string the uniquely identifies the contents of the archive, i.e.
the version. It has the same restrictions as the "exe_id", and
should be fixed-length, as Urlader assumes lexicographically higher
versions are newer, and thus preferable.
pack file (archive)
This contains the "exe_id", the "exe_ver", a number of environment
variable assignments, the program name to execute, the initial
arguments it receives, and finally, a list of files (with contents
:) and directories.
override
When the urlader starts, it first finds out what "exe_id" is
embedded in it. It then looks for an override file for this id
($URLADER_EXE_DIR/override) and verifies that it is for the same
"exe_id", and the version is newer. If this is the case, then it
will unpack and run the override file instead of unpacking the files
attched to itself.
This way one can implement software upgrades - download a new
executable, write it safely to disk and move it to the override
path.
ENVIRONMENT VARIABLES
The urlader sets and maintains the following environment variables, in
addition to any variables specified on the commandline. The values in
parentheses are typical (but not gauranteed) values for unix - on
windows, ~/.urlader is replaced by %AppData%/urlader.
URLADER_VERSION (1.0)
Set to the version of the urlader binary itself. All versions with
the same major number should be compatible to older versions with
the same major number.
URLADER_DATADIR (~/.urlader)
The data directory used to store whatever urlader needs to store.
URLADER_CURRDIR
This is set to the full path of the current working directory where
the urlader was started. Atfer unpacking, the urlader changes to the
"URLADER_EXECDIR", so any relative paths should be resolved via this
path.
URLADER_EXEPATH
This is set to the path of the urlader executable itself, usually
relative to $URLADER_CURRDIR.
URLADER_EXE_ID
This stores the executable id of the pack file attached to the
urlader.
URLADER_EXE_VER
This is the executable version of the pack file attached to the
urlader, or the override, whichever was newer. Or in other words,
this is the version of the application running at the moment.
URLADER_EXE_DIR (~/.urlader/$URLADER_EXE_ID>
The directory where urlader stores files related to the executable
with the given id.
URLADER_EXECDIR (~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER)
The directory where the files from the pack file are unpacked and
the program is being run. Also the working directory of the program
when it is run.
URLADER_OVERRIDE (empty or override)
The override file used, if any, relative to $URLADER_EXECDIR. This
is either missing, when no override was used, or the string
override, as thta is currently the only override file urlader is
looking for.
FUNCTIONS AND VARIABLES IN THIS MODULE
$Urlader::URLADER_VERSION
Set to the urlader version ("URLADER_VERSION") when the program is
running form within urlader, undef otherwise.
$Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER,
$Urlader::EXE_DIR, $Urlader::EXECDIR
Contain the same value as the environment variable of the (almost)
same name. You should prefer these, though, as these might even be
set to correct values when not running form within an urlader
environment.
Urlader::set_exe_info $exe_id, $exe_ver
Sets up the paths and variables as if running the given executable
and version from within urlader.
$lock = Urlader::lock $path, $exclusive, $wait
Tries to acquire a lock on the given path (which must specify a file
which will be created if necessary). If $exclusive is true, then it
tries to acquire an exclusive lock, otherwise the lock will be
shared. If $wait is true, then it will wait until the lock can be
acquired, otherwise it only attempts to acquire it and returns
immediately if it can't.
If successful it returns a lock object - the lock will be given up
when the lock object is destroyed or when the process exits (even on
a crash) and has a good chance of working on network drives as well.
If the lock could not be acquired, "undef" is returned.
This function is provided to assist applications that want to clean
up old versions, see "TIPS AND TRICKS", below.
TIPS AND TRICKS
Gathering files
Gathering all the files needed for distribution can be a big
problem. Right now, Urlader does not assist you in this task in any
way, however, just like perl source stripping, it is planned to
unbundle the relevant technology from staticperl
(<http://staticperl.plan9.de>) for use with this module.
You could always use par to find all library files, unpack the
bundle and add perl, libperl and other support libraries (e.g.
libgcc_s).
Software update
Updating the software can be done by downloading a new packfile
(with the same "exe_id" but a higher "exe_ver" - this can simply be
the executable you create when making a release) and replacing the
override file in the $URLADER_EXE_DIR.
When looking for updates, you should include $URLADER_VERSION,
$URLADER_EXE_ID and $URLADER_EXE_VER - the first two must be
identical for update and currently running program, while the last
one should be lexicographically higher.
Replacing the override file can be done like this:
rename "new-override.tmp", "$Urlader::EXE_DIR/override"
or die "could not replace override";
This can fail on windows when another urlader currently reads it,
but should work on all platforms even when other urlader programs
execute concurrently.
Cleaning up old directories
Urlader only packs executables once and then caches them in the
$URLADER_EXECDIR. After upgrades there will be old versions in there
that are not being used anymore. Or are they?
Each instance directory (i-*) in the $URLADER_EXE_DIR) has an
associated lock file (i-*.lck) - while urlader executes an app it
keeps a shared lock on this file.
To detect whether a version is in use or not, you must try to
acquire an exclusive lock, i.e.:
my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0;
if (!$lock) {
# instance dir is not in use and can be safely deleted
}
If an older urlader wants to use an instance that was deleted or is
currently being deleted it will wait until it's gone and simply
recreate it, so while less efficient, deleting instance directories
should always be safe.
The lockfile itself can be deleted as long as you have an exclusive
lock on it (if your platform allows this).
A real world project
The only real world project using this that I know of at the moment
is the deliantra client (http://www.deliantra.net for more info).
It uses some scary scripts to build the client and some dependnet
modules (build.*), to gather perl source files into a distribution
tree, shared objects and system shared libraries (some of which have
to be patched or, due to the horrible dll hell on OS X, even
renamed), called "gatherer", and a script called "gendist" to build
executable distributions.
These can be found at
<http://cvs.schmorp.de/deliantra/Deliantra-Client/util/>, but
looking at them can lead to premature blindless.
Shared Libraries
It is often desirable to package shared libraries - for example the
Deliantra client packages SD>, Berkely DB, Pango and amny other
libraries that are unlikely to be available on the target system.
This usually requires some fiddling (see below), and additionally
some environment variables to be set.
For example, on ELF systems you usually want LD_LIBRARY_PATH=. and
on OS X, you want DYLD_LIBRARY_PATH=. (these are effectively the
default on windows).
These can most easily be specified when building the packfile:
urlader-util ... LD_LIBRARY_PATH=. ./perl run
Portability: RPATH
Often perl is linked against a shared libperl.so - and might be so
using an rpath. Perl extensikns likewise might use an rpath, which
means the binary will mostly ignore LD_LIBRARY_PATH, which leads to
trouble.
There is an utility called chrpath, whose -d option can remove the
rpath from binaries, shared library and shared objects.
Portability: OS X DLL HELL
OS X has the most severe form of DLL hell I have seen - if you link
against system libraries, which is practically unavoidable, you get
libraries of well-known names (e.g. libjpeg) that have nothing to do
with what you normally expect libjpeg to be, and there is no way to
get your version of libjpeg into your program.
Moreover, even if apple ships well-known libraries (e.g. libiconv),
they often ship patched versions which have a different ABI or even
API then the real releases.
The only way aorund this I found was to change all library names in
my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and so
on), by patching the paths in the share dlibraries and shared
objects. install-name-tool (with -id and -change) works in many
cases, but often paths are embedded indirectly, so you might have to
use a *dirty* string replacement.
SECURITY CONSIDERATIONS
The urlader executable itself does not support setuig/setgid operation,
or running with elevated privileges - it does no input sanitisation, and
is trivially exploitable.
AUTHOR
Marc Lehmann <schmorp@schmorp.de>
http://home.schmorp.de/