David Yingling > App-Fetchware-1.010 > App::Fetchware

Download:
App-Fetchware-1.010.tar.gz

Dependencies

Annotate this POD

View/Report Bugs
Module Version: 1.010   Source  

NAME ^

App::Fetchware - App::Fetchware is Fetchware's API used to make extensions.

VERSION ^

version 1.010

SYNOPSIS ^

    ### App::Fetchware's use inside a Fetchwarefile.
    ### See fetchware's new command for an easy way to create Fetchwarefiles.
    use App::Fetchware;

    # Only program, lookup_url, one or more mirrors, and some method of
    # verification are required.
    program 'Your program';
    lookup_url 'http://whatevermirror.your/program/is/on';
    gpg_keys_url 'http://whatevermirror.your/program/gpg/key/url.asc';
    mirror 'http://whatevermirror1.your/program/is/on';
    mirror 'http://whatevermirror2.your/program/is/on';
    mirror 'http://whatevermirror3.your/program/is/on';
    mirror 'http://whatevermirror4.your/program/is/on';
    mirror 'http://whatevermirror5.your/program/is/on';

    # filter is not required, but is often needed to tell fetchware which
    # program in the lookup_url directory or what specific version you would
    # want to install. For example, Apache maintains 3 versions 2.0, 2.2, and
    # 2.4. filter is what allows you to select which version you want fetchware
    # to use.
    filter 'version-2.0';

    # Below are some popular options that may interest you.
    make_options '-j 4';

    ### This is how Fetchwarefile's can replace lookup()'s or any other
    ### App::Fetchware API subroutine's default behavior.
    ### Remember your coderef must take the same parameters and return the same
    ### values.
    hook lookup => sub {
        # Callback that replaces lookup()'s behavior.
        # Callback receives the same arguments as lookup(), and is must return
        # the same number and type of arguments that lookup() returns.
        return $download_path;
    };


    ### See EXTENDING App::Fetchware WITH A MODULE for details on how to extend
    ### fetchware with a module to install software that cannot be expressed
    ### using App::Fetchware's configuration file syntax.

DESCRIPTION ^

App::Fetchware represents fetchware's API. For ducumentation on how to use App::Fetchware's fetchware command line interface see fetchware.

It is the heart and soul of fetchware where all of fetchware's main behaviors are kept. It is fetchware's API, which consists of the subroutines start(), lookup(), download(), verify(), unarchive(), build(), install(), uninstall(), and end().

App::Fetchware stores both details about fetchware's configuration file syntax, documents how to create a fetchware extension, and documents the internal workings of how App::Fetchware implements fetchware's package management behavior:

CREATING A App::Fetchware FETCHWAREFILE ^

In order to create a new fetchware package, you need to create a new Fetchwarefile. You can easily do this with the fetchware new command, which works as follows.

1. This command will ask you a few questions, and use the answers you provide to create a Fetchwarefile for you.
2. After it does so, it gives you a chance to edit its autogenerated Fetchwarefile manually in an editor of your choice.
3. Afterwards, it will ask you if you would like to go ahead and use your newly created Fetchwarefile to install your new program as a fetchware package. If you answer yes, the default, it will install it, but if you anwer no; instead, it will simply print out the location to the Fetchwarefile that it created for you. You can then copy that file to a location of your choice, or use that path as an option to additional fetchware commands.

You can also create your Fetchwarefile manually in a text editor if you want to. See the section "MANUALLY CREATING A App::Fetchware FETCHWAREFILE" for the details. Some programs require greater customization of Fetchware's behavior than is available in its configuration options in these cases see the section "FURTHER CUSTOMIZTING YOUR FETCHWAREFILE" for the specific details on how to make fetchware do what it needs to do to manage your source code distributions.

MANUALLY CREATING A App::Fetchware FETCHWAREFILE ^

fetchware provides a new command that allows you to easily create a Fetchwarefile by simply answering a bunch of simple questions. This new command even will let you manually edit the Fetchwarefile it generates for you, so you can later customize it however you want to. Then it will ask if you want to install it. If you answer yes, then it will install it for you, if you answer no, it will print out the path to the newly created Fetchwarefile it created for you. Then you can call fetchware install path/to/Fetchwarefile to install it later on. Or you can follow the instructions below and manually create a Fetchwarefile in your text editor of choice.

Fetchware's Fetchwarefile's configuration option syntax

The syntax for setting configuration options is easy. It's just the name of the configuration option you want to specify like so:

    program

And then you add a space, and then whatever value you want it to have in quotes.

    program 'Apache';

And don't forget the semicolon ; on then end. The semicolon is required

You can use comments as needed to help document you Fetchwarefile like so:

    # Fetchwarefile for Apache.
    program 'Apache';
1. Name it

Use your text editor to create a file with a .Fetchwarefile file extension. Use of this convention is not required, but it makes it obvious what type of file it is. Then, just copy and paste the example text below, and replace [program] with what you choose the name of your proram to be. program is simply a configuration option that simply names your Fetchwarefile. It is not actually used for anything other than to name your Fetchwarefile to document what program or behavior this Fetchwarefile manages.

    use App::Fetchware;

    # [program] - explain what [program] does.
    program '[program]';

Fetchwarefiles are actually small, well structured, Perl programs that can contain arbitrary perl code to customize fetchware's behavior, or, in most cases, simply specify a number of fetchware or a fetchware extension's configuration options. Below is my filled in example App::Fetchware fetchwarefile.

    use App::Fetchware;

    # apache 2.2 - Web Server.
    program 'apache-2.2';

Notice the use App::Fetchware; line at the top. That line is absolutely critical for this Fetchwarefile to work properly, because it is what allows fetchware to use Perl's own syntax as a nice easy to use syntax for Fetchwarefiles. If you do not use the matching use App::Fetchware; line, then fetchware will spit out crazy errors from Perl's own compiler listing all of the syntax errors you have. If you ever receive that error, just ensure you have the correct use App::Fetchware; line at the top of your Fetchwarefile.

2. Determine your lookup_url

At the heart of App::Fetchware is its lookup_url, which is the URL to the FTP or HTTP mirror you want App::Fetchware to use to obtain a directory listing to see if a new version of your program is available for download. To figure this out just use your browser to find the program you want fetchware to manage for you's Web site. Skip over the download link, and instead look for the gpg, sha1, or md5 verify links, and copy and paste one of those between the single quotes above in the lookup_url. Then delete the file portion--from right to left until you reach a /. This is necessary, because fetchware uses the lookup_url as a basis to download your the gpg, sha1, or md5 digital signatures or checksums to ensure that the packages fetchware downloads and installs are exactly the same as the ones the author uploads.

    lookup_url '';

And then after you copy the url.

    lookup_url 'http://www.apache.org/dist/httpd/';
3. Determine your filter configuration option

The filter option specifies a perl regex that is matched against the list of the files in the directory you specify in your lookup_url. This sorts through the directory to pick out which program or even which version of the same program you want this Fetchwarefile to manage.

This is needed, because some programs such as Apache have multiple versions available at the same time, so you would need to specify which version of apache you want to download. So, you'd specify filter 'httpd-2.2'; to make fetchware download and manage Apache 2.2, or you could specify filter 'httpd-2.4'; to specify the newer 2.4 series version.

This option also exists to allow fetchware to pick out our program if you specify a mirror directory that has more than one program in it. If you do this, then fetchware can use filter to pick out the program you want to download and install from the crowd.

Just write your perl regex in between the single quotes ' below. You don't need to master regular expressions to specify this option. Just specify the name of the program and/or the main part of the version number. Do not specify the entire version number or fetchware will never update your program properly.

    filter '';

And then after you type in the text pattern.

    filter 'httpd-2.2';
4. Add mandatory verification settings

Verification of software downloads is mandatory, because fetchware, in order to install the software that is downloaded, must execute the build and installation scripts on your computer sometimes even as the root administrator! Therefore, fetchware will refuse to build and install any software package that cannot be verified. This limitation can be bypassed by setting the verify_failure_ok configuration option to true, but his is not recommended.

Instead, if standard verification fails, please set up one or more of the configuration options below that may allow verification to fail if the author has his download site set up differently then fetchware expects.

gpg_keys_url - Should list a URL to a file most likely named KEYS that contains versions of the author's gpg verification keys that is suitable to be imported into gpg using gpg --import [name of file]. An example would be:
    gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';
users_keyring - Tells fetchware to use the user who calls fetchware's gpg keyring instead of fetchware's own keyring. This is handy for when you want to install a program, but the author has no easily accessible KEYS file, but the author has listed his gpg key on his Website. With this option, you can import this key into your own keyring using gpg --import [name of file], and then specify this option in your Fetchwarefile as shown below.
    users_keyring 'On';
gpg_sig_url - Should list a URL to a directory (not a file) that has files with the same names as the software archives that contain your program, but with a .asc, .sig, or .sign file extension. An example would be:
    gpg_sig_url 'http://www.apache.org/dist/httpd/';
sha1_url - Should list a URL to a directory (not a file) that has files with the same names as the software archives that contain your program, but with a .sha or .sha1 file extension. An example would be:
    sha1_url 'http://www.apache.org/dist/httpd/';
md5_url - Should list a URL to a directory (not a file) that has files with the same names as the software archives that contain your program, but with a .md5 file extension. An example would be:
    md5_url 'http://www.apache.org/dist/httpd/';
NOTICE: There is no configuration option to change what filename fetchware uses. You're stuck with its default of what fetchware determines your $download_path to be with the appropriate .asc, sha1, or .md5 added to it.

Just copy and paste the example above replacing the example between the single quotes ' with the actual value you need.

5. Specify at least one mirror

Because fetchware's lookup_url must be the author's main mirror instead of a 3rd party mirror for verification purposes, you must also add a mirror option that specifies one 3rd party mirror. I recommend picking one near your physical geographical location or at least in your own country or one close by.

mirror can be specified more than once, you you can have more than one mirror. An example is below.

    mirror 'http://apache.mesi.com.ar//httpd/';
    mirror 'http://apache.osuosl.org//httpd/';
    mirror 'ftp://apache.mirrors.pair.com//httpd/';
    mirror 'http://mirrors.sonic.net/apache//httpd/';
    mirror 'http://apache.mirrors.lucidnetworks.net//';

You can specify as many mirrors as you want to. You could perhaps include all the mirrors your source code distribution has. And the mirrors are tried in the order they are specified in your Fetchwarefile.

6. Specifiy other options

That's all there is to it unless you need to further customize App::Fetchware's behavior to modify how your program is installed.

If your Fetchwarefile is now finished, you can install your new Fetchwarefile as a fetchware package with:

    fetchware install [path to your new fetchwarefile]

Or you can futher customize it further as shown next if needed.

7. Optionally add build and install settings

If you want to specify additional settings the first to choose from are the build and install settings. These settings control how fetchware builds and installs your software. They are briefly listed below. For further details see the section "App::Fetchware FETCHWAREFILE CONFIGURATION OPTIONS".

temp_dir - Specifies the temporary directory fetchware will use to create its own working temporary directory where it downloads, unarchives, builds, and then installs your program from a directory inside this directory.
user - (UNIX only) - Specifies a non-root user to drop privileges to when downloading, verifying, unarchive, and building your program. Root priveedges are kept in the parent process for install if needed.
prefix - Specifies the --prefix option for AutoTools (./configure) based programs.
configure_options - Specifies any additional options that fetchware should give to AutoTools when it runs ./configure to configure your program before it is built and installed.
make_options - Specifies any command line options you would like to provide to make when make is run to build and install your software. -j 4 is quite popular to do a paralled make to build and install the program faster.
build_commands - Specifies a list of commands that fetchware will use to build your program. You only need this option if your program uses a build system other than AutoTools such as cmake or perhaps a custom one like Perl's Configure
install_commands - Specifies a list of commands that fetchware will use to install your program. You only need this option if your program uses a build system other than AutoTools such as cmake or perhaps a custom one like Perl's Configure
uninstall_commands - Specifies a list of commands that fetchware will use to uninstall your program. You only need this option if your source code distribution does not provide a make uninstall target, which not every source code distribution does.
no_install - Specifies a boolean (true or false) value to turn off fetchware installing the software it has downloaded, verified, unarchvied, and built. If you specify a true argument (1 or 'True' or 'On'), then fetchware will not install your program; instead, it will leave its temporary directory intact, and print out the path of this directory for you to inspect and install yourself. If you don't specify this argument, comment it out, or provide a false argument (0 or 'False' or 'Off'), then fetchware will install your program.

Just copy and paste the example below replacing [new_directive] with the name of the new directive you would like to add, and fill in the space between the single quotes '.

    [new_directive] '';

After pasting it should look like.

    [new_directive] '~/wallpapers';

USING YOUR App::Fetchware FETCHWAREFILE WITH FETCHWARE ^

After you have created your Fetchwarefile as shown above you need to actually use the fetchware command line program to install, upgrade, or uninstall your App::Fetchware Fetchwarefile.

install

A fetchware install [path/to/Fetchwarefile] while using a App::Fetchware Fetchwarefile causes fetchware to install the program specified in your fetchwarefile to your computer as you have specified any build or install options.

upgrade

A fetchware upgrade [installed program name] while using a App::Fetchware Fetchwarefile will simply run the same thing as install all over again, which ill upgrade your program if a new version is available.

uninstall

A fetchware uninstall [installed program name] will cause fetchware to run the command make uninstall, or run the commands specified by the uninstall_commands configuration option. make uninstall is only available from some programs that use AutoTools such as ctags, but apache, for example, also uses AutoTools, but does not provide a uninstall make target. Apache for example, therefore, cannot be uninstalled by fetchware automatically.

upgrade-all

A fetchware upgrade-all will cause fetchware to run fetchware upgrade for all installed packages that fetchware is tracking in its internal fetchware database. This command can be used to have fetchware upgrade all currently installed programs that fetchware installed.

If you would like fetchware upgrade-all to be run every night automatically by cron, then just create a file say fetchware with the contents below in it, and add it to /etc/cron.daily.

    #!/bin/sh
    # Update all already installed fetchware packages.
    fetchware upgrade-all

And if you don't want to run it system wide as root, you can add it to your user crontab by pasting the snippet below in to your crontab by executing crontab -e.

    # Check for updates using fetchware every night at 2:30AM.
    # Minute   Hour   Day of Month     Month          Day of Week     Command    
    # (0-59)  (0-23)     (1-31)  (1-12 or Jan-Dec) (0-6 or Sun-Sat)
        30      2          *              *               *           fetchware upgrade-all

App::Fetchware'S FETCHWAREFILE CONFIGURATION OPTIONS ^

App::Fetchware has many configuration options. Most were briefly described in the section "MANUALLY CREATING A App::Fetchware FETCHWAREFILE". All of them are detailed below.

program 'Program Name';

program simply gives this Fetchwarefile a name. It is availabe to fetchware after parsing your Fetchwarefile, and is used to name your Fetchwarefile when using fetchware new. It is required just like lookup_url, mirror, perhaps filter, and some method to verify downloads are.

filter 'perl regex here';

Specifies a Perl regular expression that fetchware uses when it determines what the latest version of a program is. It simply compares each file in the directory listing specified in your lookup_url to this regular expression, and only matching files are allowed to pass through to the next part of fetchware that looks for source code archives to download.

See perlretut for details on how to use and create Perl regular expressions; however, actual regex know how is not really needed just paste verbatim text between the single quotes '. For example, filter 'httpd-2.2'; will cause fetchware to only download Apache 2.2 instead of the version for Windows or whatever is in the weird httpd-deps-* package.

temp_dir '/tmp';

temp_dir tells fetchware where to store fetchware's temporary working directory that it uses to download, verify, unarchive, build, and install your software. By default it uses your system temp directory, which is whatever directory File::Temp's tempdir() decides to use, which is whatever File::Spec's tmpdir() decides to use.

fetchware_db_path '~/.fetchwaredb';

fetchware_db_path tells fetchware to use a different directory other than its default directory to store the installed fetchware package for the particular fetchware package that this option is specified in your Fetchwarefile. Fetchware's default is /var/log/fetchware on Unix when run as root, and something like /home/[username]/.local/share/Perl/dist/fetchware/ when run nonroot.

This option is not recommended unless you only want to change it for just one fetchware package, because fetchware also consults the FETCHWARE_DATABASE_PATH environment variable that you should set in your shell startup files if you want to change this globally for all of your fetchware packages. For sh/bash like shells use:

    export FETCHWARE_DATABASE_PATH='/your/path/here'

user 'nobody';

Tells fetchware what user it should drop privileges to. The default is nobody, but you can specify a different username with this configuration option if you would like to.

Dropping privileges allows fetchware to avoid downloading files and executing anything inside the downloaded archive as root. Except of course the commands needed to install the software, which will still need root to able to write to system directories. This improves security, because the downloaded software won't have sytem privileges until after it is verified, providing that what you downloaded is exactly what the author uploaded.

Note this only works for unix like systems, and is not used on Windows and other non-unix systems.

Also note, that if you are running fetchware on Unix even if you do not specify the user configuration option to configure what user you will drop privileges to, fetchware will still drop privileges using the ubiquitous nobody user. If you do not want to drop privileges, then you must use the stay_root configuration option as described below.

stay_root 'On';

Tells fetchware to not drop privileges. Dropping privileges when run as root is fetchware's default behavior. It improves security, and allows fetchware to avoid exposing the root account by downloading files as root.

Do not use this feature unless you are absolutely sure you need it.

SECURITY NOTICE

stay_root, when turned on, causes fetchware to not drop privileges when fetchware looks up, downloads, verifies, and builds your program. Instead, fetchware will stay root through the entire build cycle, which needlessly exposes the root account when downloading files from the internet. These files may come from trusted mirrors, but mirrors can, and do get cracked:

http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0

http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html

http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans

http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys

lookup_url 'ftp://somedomain.com/some/path

This configuration option specifies a url of a FTP or HTTP or local (file://) directory listing that fetchware can download, and use to determine what actual file to download and perhaps also what version of that program to download if more than one version is available as some mirrors delete old versions and only keep the latest one.

This url is used for:

1. To determine what the actual download url is for the latest version of this program
2. As the base url to also download a cryptographic signature (ends in .asc) or a SHA-1 or MD5 signature to verify the contents match what the SHA-1 or MD5 checksum is.

You can use the mirror configuration option to specify additional mirrors. However, those mirrors will only be used to download the large software archives. Only the lookup_url will be used to download directory listings to check for new versions, and to download digital signatures or checksums to verify downloads.

lookup_method 'timestamp';

Fetchware has two algorithms it uses to determine what version of your program to download:

timestamp

The timestamp algorithm simply uses the mtime (last modification time) that is availabe in FTP and HTTP directory listings to determine what file in the directory is the newest. timestamp is also the default option, and is the one used if lookup_method is not specified.

versionstring

Versionstring parses out the version numbers that each downloadable program has, and uses them to determine the downloadable archive with the highest version number, which should also be the newest and best version of the archive to use.

gpg_keys_url 'lookup_url.com/some/path';

Specifies a file not a directory URL for a KEYS file that lists all of the authors' gpg keys for fetchware to download and import before using them to verify the downloaded software package.

If you come accross a software package whoose author uses gpg to sign his software packages, but he does not include it in the form of a file on his main mirror, then you can specify the user_keyring option. This option forces fetchware to use the user who runs fetchware's keyring instead of fetchware's own keyring. This way you can then import the author's key into your own keyring, and have fetchware use that keyring that already has the author's key in it to verify your downloads.

user_keyring 'On';

When enabled fetchware will use the user who runs fetchware's keyring instead of fetchware's own keyring. Fetchware uses its own keyring to avoid adding cruft to your own keyring.

This is needed when the author of a software package does not maintain a KEYS file that can easily be downloaded and imported into gpg. This option allows you to import the author's key manually into your own gpg keyring, and then fetchware will use your own keyring instead of its own to verify your downloads.

LIMITAITON

user_keyring when set to true requires that the user that fetchware is running under have a real gpg keyring with keys that have been imported into it. This is not the case unless the user option has been specified with a user account with a proper home directory and gpg keyring for gpg to use. Because of this limitation if you need to specify user_keyring be sure to also specify the user option to specify a real user account instead of the default fake one nobody.

Typically you would import the keys into your own user accounts gpg keyring, and then you would specify your own username with the user option to tell fetchware to drop privs to your own user account to have access to your own gpg keys.

gpg_sig_url 'mirror.com/some/path';

Specifies an alternate url to use to download the cryptographic signature that goes with your program. This is usually a file with the same name as the download url with a .asc file extension added on. Fetchware will also append the extensions sig and sign if .asc is not found, because some pgp programs and authors use these extensions too.

sha1_url 'mastermirror.com/some/path';

Specifies an alternate url to download the SHA-1 checksum. This checksum is used to verify the integrity of the archive that fetchware downloads.

You must specify the master mirror site, which is your programs main mirror site, because if you download it from a mirror, its possible that both the archive and the checksum could have been tampered with.

md5_url 'mastermirror.com/some/path';

Specifies an alternate url to download the MD5 checksum. This checksum is used to verify the integrity of the archive that fetchware downloads.

You must specify the master mirror site, which is your programs main mirror site, because if you download it from a mirror, its possible that both the archive and the checksum could have been tampered with.

user_agent 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0'

Specifies what user_agent you would like fetchware to pretend to be when downloading files using the HTTP protocol. Some sites annoying prevent some user agents from working while allowing others. This allows you to pretend to be a real browser such as Firefox if you need to.

verify_method 'gpg';

Chooses a method to verify your program. The default is to try gpg, then sha1, and finally md5, and if all three fail, then the default is to exit fetchware with an error message, because it is insecure to install archives that cannot be verified. The availabel options are:

gpg - Uses the gpg program to cryptographically verify that the program you downloaded is exactly the same as its author uploaded it.
sha1 - Uses the SHA-1 hash function to verify the integrity of the download. This is much less secure than gpg.
md5 - Uses the MD5 hash function to verify the integrity of the download. This is much less secure than gpg.

verify_failure_ok 'True';

Fetchware's default regarding failing to verify your downloaded Archive with gpg, sha1, or md5 is to exit with an error message, because installing software that cannot be cryptographically verified should never be done.

SECURITY NOTICE

However, if the author of a program you want to use fetchware to manage for you does not offer a gpg, sha1, or md5 file to verify its integrity, then you can use this option to force Fetchware to install this program anyway. However, do not enable this option lightly. Please scour the program's mirrors and homepage to see which gpg_keys_url, gpg_sig_url, sha1_url, md5_url, or user_keyring you can use to ensure that your archive is verified before it is compiled and installed. Even mirrors from sites large and small get hacked regularly:

http://www.itworld.com/security/322169/piwik-software-installer-rigged-back-door-following-website-compromise?page=0,0

http://www.networkworld.com/news/2012/092612-compromised-sourceforge-mirror-distributes-backdoored-262815.html

http://www.csoonline.com/article/685037/wordpress-warns-server-admins-of-trojans

http://www.computerworld.com/s/article/9233822/Hackers_break_into_two_FreeBSD_Project_servers_using_stolen_SSH_keys

So, Please give searching for a gpg_keys_url, gpg_sig_url, sha1_url, md5_url, or user_keyring for your program another try before simply enabling this option.

NOTICE

verify_failure_ok is a boolean configuration option, which just means its values are limited to either true or false. True values are 'True', 'On', 1, and false values are 'False', 'Off', and 0. All other values are syntax errors.

prefix '/opt/';

Controls the AutoTools ./configuration --prefix=... option, which allows you to change the base directory that most software (software that uses AutoTools) uses as the base directory for when they install themselves.

For example, most programs copy binaries to prefix/bin, documentation to prefix/docs, manpages to prefix/man, and so on.

WARNING: prefix only supports source code distributions that use GNU AutoTools. These can easily be determined by the presence of certain files in the the distributions main directory such as configure, configure.in, and acinclude.m4, and others. So, if your program uses a different build system just include that system's version of AutoTools' --prefix option in your build_commands configuration option.

configure_options '--datadir=/var/mysql';

Provides options to AutoTools ./configure program that configures the source code for building. Most programs don't need this, but some like Apache and MySQL need lots of options to configure them properly. In order to provide multiple options do not separate them with spaces; instead, separate them with commas and keep single quotes ' around them like in the example below.

    configure_options '--datadir=/var/mysql', '--mandir=/opt/man',
        '--enable-module=example';

This option is not compatible with build_commands. If you use build_commands, than this option will not be used.

WARNING: configure_options only supports source code distributions that use GNU AutoTools. These can easily be determined by the presence of certain files in the the distributions main directory such as configure, configure.in, and acinclude.m4, and others. So, if your program uses a different build system just include that system's version of AutoTools' ./configure program in your build_commands configuration option.

make_options '-j4';

This option exists mostly just to enable parallel make using the -j jobs option. But any list of options make accepts will work here too. Separate them using commas and surround each one with single quotes ' like in the example above.

build_commands './configure', 'make';

Specifies what commands fetchware will use to build your program. Building your program includes configuring the source code to be compiled, and then the actual compiling too. It does not include installing the compiled source code. Use install_commands, which is described below, for that. Specify multiple options just like configure_options does.

The default build_commands is simply to run ./configure followed by make.

install_commands 'make install';

This single command or perhaps list of commands will be run in the order your specifyto install your program. You can specify multiple options just like configure_options does.

The default install_commands is simply to run make install.

uninstall_commands 'make uninstall';

This command or list of commands will be run instead of fetchware's default of make uninstall. Many source code distributions do not provide a uninstall make target, so they can not easily be uninstalled by fetchware without such support. In these cases, you could look into paco, src2pkg, or fpm. These all aid turning a source code distribution into your operating system's package format, or somehow magically monitoring make install to track what files are installed where, and then using this information to be able to uninstall them.

no_install 'On';

This boolean, see below, configuration option determines if fetchware should install your software or not install your software, but instead prints out the path of its build directory, so that you can QA test or review the software before you install it.

NOTICE

no_install is a boolean configuration option, which just means its values are limited to either true or false. True values are 'True', 'On', 1, and false values are 'False', 'Off', and 0. All other values are syntax errors.

mirror 'somemirror0.com/some/optional/path';

Your Fetchwarefile needs to have at least one mirror specified. Although you can specify as many as you want to.

This configuration option, unlike all the others, can be specified more than once. So, for example you could put:

    mirror 'somemirror1.com';
    mirror 'somemirror2.com';
    mirror 'somemirror3.com';
    mirror 'somemirror4.com';
    mirror 'somemirror5.com';

When fetchware downloads files or directories it will try each one of these mirrors in order, and only fail if all attempts at all mirrors fail.

If you specify a path in addition to just the hostname, then fetchware will try to get whatever it wants to download at that alternate path as well.

    mirror 'somemirror6./com/alternate/path';

FURTHER CUSTOMIZING YOUR FETCHWAREFILE ^

Because fetchware's configuration files, its Fetchwarefiles, are little Perl programs, you have the full power of Perl at your disposal to customize fetchware's behavior to match what you need fetchware to do to install your source code distributions.

Not only can you use arbitrary Perl code in your Fetchwarefile to customize fetchware for programs that don't follow most FOSS mirroring's unwritten standards or use a totally different build system, you can also create a fetchware extension. Creating a fetchware extension even allows you to turn your extension into a proper CPAN distribution, and upload it to CPAN to share it with everybody else. See the section below, "CREATING A FETCHWARE EXTENSION", for full details.

How Fetchware's configuration options are made

Each configuration option is created with App::Fetchware::CreateConfigOptions This package's import() is a simple code generator that generates configuration subroutines. These subroutines have the same names as fetchware's configuration options, because that is exactly what they are. Perl's Prototypes are used in the code that is generated, so that you can remove the parentheses typically required around each configuration subroutine. This turns what looks like a function call into what could believably be considered a configuration file syntax.

These prototypes turn:

    lookup_url('http://somemirror.com/some/path');

Into:

    lookup_url 'http://somemirror.com/some/path';

Perl's prototypes are not perfect. The single quotes and semicolon are still required, but the lack of parens instantly makes it look much more like a configuration file syntax, then an actual programming language.

The magic of use App::Fetchware;

The real magic power behind turning a Perl program into a configuration file sytax comes from the use App::Fetchware; line. This line is single handedly responsible for making this work. This one line imports all of the configuration subroutines that make up fetchware's configuration file syntax. And this mechanism is also behind fetchware's extension mechanism. (To use a App::Fetchware extension, you just use it. Like use App::FetchwareX::HTMLPageSync;. That's all there is to it. This other App::Fetchware is responsible for exporting subroutines of the same names as those that make up App::Fetchware's API. These subroutines are listed in the section "FETCHWAREFILE API SUBROUTINES" as well as their helper subroutines. See the section below "CREATING A FETCHWARE EXTENSION" for more information on how to create App::Fetchware extensions.

So how do I add some custom Perl code to customize my Fetchwarefile?

You use hook() to override one of fetchware's API subroutines. Then when fetchware goes to call that subroutine, your own subroutine is called in its place. You can hook() as many of fetchware's API subroutines as you need to.

Remember your replackement subroutine must take the exact same arguments, and return the same outputs that the standard fetchware API subroutines do!

All of the things these subroutines return are later used as parameters to later API subroutines, so failing to return a correct value may cause fetchware to fail.

hook()

    # Inside a Fetchwarefile...
    hook lookup => sub {
        # Your own custom lookup handler goes here!
    };

hook() allows you to replace fetchware's API subroutines with whatever Perl code reference you want to. But it must take the same arguments that each API subroutine takes, and provide the same return value. See the section "FETCHWAREFILE API SUBROUTINES" for the details of what the API subroutine's parameters are, and what their return values should be.

hook() should be used sparingly, and only if you really know what you're doing, because it directly changes fetchware's behavior. It exists for cases where you have a software package that exceeds the abilities of fetchware's configuration options, but creating a fetchware extension for it would be crazy overkill.

    hook lookup => sub {
        # Your replacement for lookup() goes here.
    };

However, that is not quite right, because some of App::Fetchware's API subroutines take important arguments and return important arguments that are then passed to other API subroutines later on. So, your replacement lookup() must take the same arguments and must return the same values that the other App::Fetchware subroutines may expect to be passed to them. So, let's fix lookup(). Just check lookup()'s documentation to see what its arguments are and what it returns by checking out the section "FETCHWAREFILE API SUBROUTINES":

    hook lookup => sub {
        # lookup does not take any arguments.
        
        # Your replacement for lookup() goes here.
        
        # Must return the same thing that the original lookup() does, so
        # download() and everything else works the same way.
        return $download_path;
    };

Some App::Fetchware API subroutines take arguments, so be sure to account for them:

    hook download => sub {
        # Take same args as App::Fetchware's download() does.
        my $download_path = shift;
        
        # Your replacement for download() goes here.
        
        # Must return the same things as App::Fetchware's download()
        return $package_path;
    };

If changing lookup()'s behavior or one of the other App::Fetchware subroutines, and you only want to change part of its behavior, then consider importing one of the :OVERRIDE_* export tags. These tags exist for most of the App::Fetchware API subroutines, and are listed below along with what helper subroutines they import with them. To check their documentation see the section "FETCHWAREFILE API SUBROUTINES".

WARNING

If you specify a OVERRIDE_* export tag to App::Fetchware be sure to add the :DEFAULT export tag to also export App::Fetchware's default exports, which must be properly exported for fetchware to work properly.

OVERRIDE_LOOKUP - "check_lookup_config()", "get_directory_listing()", "parse_directory_listing()", "determine_download_path()", "ftp_parse_filelist()", "http_parse_filelist()", "file_parse_filelist()", "lookup_by_timestamp()", "lookup_by_versionstring()", "lookup_determine_downloadpath()"
OVERRIDE_DOWNLOAD - "determine_package_path()"
OVERRIDE_VERIFY - "gpg_verify()", "sha1_verify()", "md5_verify()", "digest_verify()"
OVERRIDE_UNARCHIVE - "check_archive_files()", "list_files()", "list_files_tar()", "list_files_zip()", "unarchive_package()", "unarchive_tar()", "unarchive_zip()"
OVERRIDE_BUILD - "run_star_commands()" and "run_configure()".
OVERRIDE_INSTALL - "chdir_unless_already_at_path()".
OVERRIDE_UNINSTALL - uninstall() uses build()'s and install()'s API's, but does not add any subroutines of its own..

An example:

    use App::Fetchware ':OVERRIDE_LOOKUP';

    ...

    hook lookup => sub {

        ...

        # ...Download a directory listing....

        # Use same lookup alorithms that lookup() uses.
        return lookup_by_versionstring($filename_listing);
        
        # Return what lookup() needs to return.
        return $download_path;
    };

Feel free to specify a list of the specifc subroutines that you need to avoid namespace polution, or install and use Sub::Import if you demand more control over imports.

A real example

See the section "EXAMPLE FETCHWAREFILES" for real examples of using hook() to change fetchware's behavior enough to make it work properly with different source code distributions that are popular.

CREATING A FETCHWARE EXTENSION ^

WARNING

Currently, fetchware's extension system is ALPHA, and is subject to change at any time. This, however, is unlikely, but it could happen. Most likely, some new API subroutines will just be introducted such as check_syntax() to check syntax for extensions, and new(), which will allow fetchware extensions to customize the fetchware new command.

Fetchware's main program fetchware uses App::Fetchware's short and simple API to implement fetchware's default behavior; however, other styles of source code distributions exist on the internet that may not fit inside App::Fetchware's capabilities. In addition to its flexible configuration file sytax, that is why fetchware allows modules other than App::Fetchware to provide it with its behavior.

How the API works

When fetchware installs or upgrades something it executes the API subroutines start(), lookup(), download(), verify(), unarchive(), build(), install(), and end() in that order. And when fetchware uninstalls an installed package it executes the API subroutines start(), part of build(), uninstall(), and end().

Extending App::Fetchware

This API can be overridden inside a user created Fetchwarefile by using hook() as explained above. hook() simply takes a Perl code reference that takes the same parameters, and returns the same results that the subroutine that you've hook()ed takes and returns.

For more extensive changes you can create a App::Fetchware module that "subclasses" App::Fetchware. Now App::Fetchware is not an object-oriented module, so you cannot use parent or base to add it to your program's inheritance tree using @INC. You can, however, use App::Fetchware::ExportAPI to import whatever subroutines from App::Fetchware that you want to reuse such as start() and end(), and then simply implement the remaining subroutines that make up App::Fetchware's API. Just like the CODEREF extensions mentioned above, you must take the same arguments and return the same values that fetchware expects or using your App::Fetchware extension will blow up in your face.

This is described in much more detail below in "CHANGING FETCHWARE'S BEHAVIOR".

Essential Terminology

App::Fetchware manages to behave like an object oriented module would with regards to its extension system without actually using perl's object-oriented features especially using @INC for subclassing, which App::Fetchware does not use.

The same terminology as used in OOP is also used here in App::Fetchware, because the concepts are nearly the same they're simply implemented differently.

API subroutines

These are the subroutines that App::Fetchware implements, and that fetchware uses to implement its desired behavior. They are start(), lookup(), download(), verify(), unarchive(), build(), install(), uninstall(), and end(). All must be implemented in a App::Fetchware subclass.

override

Means the same thing it does in object-oriented programming. Changing the definition of a method/subroutine without changing its name. In OOP this is simply done by subclassing something, and then defining one of the methods that are in the superclass in the subclass. In App::Fetchware extensions this is done by simply defining a subroutine with the same name as one or more of the API subroutines that are defined above. There is no method resolution order and @INC is not consulted.

subclass

Means the same thing it does in object-oriented programming. Taking one class and replacing it with another class. Only since App::Fetchware is not object-oriented, it is implemented differently. You simply use App::Fetchware::ExportAPI to specify the "API subroutines" that you are not going to override, and then actually implement the remaining subroutines, so that your App::Fetchware subclass has the same interface that App::Fetchware does.

To create a fetchware extension you must understand how they work:

1. First a Fetchwarefile is created, and what module implements App:Fetchware's API is declared with a use App::Fetchware...; line. This line is use App::Fetchware for default Fetchwarefiles that use App::Fetchware to provide fetchware with the API it needs to work properly.
2. To use a fetchware extension, you simply specify the fetchware extension you want to use with a use App::Fetchware...; instead of specifying use App::Fetchware line in your Fetchwarefile. You must replace the App::Fetchware import with the extension's. Both cannot be present. Fetchware will exit with an error if you use more than one App::Fetchware line without specifying specific subroutines in all but one of them.
3. Then when fetchware parses this Fetchwarefile when you use it to install, upgrade, or uninstall something, This use App::Fetchware...; line is what imports App::Fetchware's API subroutines into fetchware's namespace.

That's all there is to it. That simple use App::Fetchware...; imports from App::Fetchware or a App::Fetchware extension such as App::FetchwareX::HTMLPageSync the API subroutines (such as start(), lookup(), ..., install(), and uninstall()) fetchware needs to use to install, upgrade, or uninstall whatever program your Fetchwarefile specifies.

After understanding how they work, simply follow the instructons and consider the recommendations below. Obviously, knowing Perl is required. A great place to start is chromatic's Modern Perl.

Develop your idea keeping in mind fetchware's package manager metaphor

Fetchware is a package manager like apt-get, yum, or slackpkg. It installs, upgrades, or uninstalls programs. Fetchware is not a Plack for command line programs. It has a rather specific API meant to fit its package manager metaphor. So, keep this in mind when developing your idea for a fetchware extension.

Map your extension's behavior to App::Fetchware's API

App::Fetchware has a specific behavior consisting of just a few subroutines with specific names that take specific arguments, and return specific values. This API is how you connect your extension to fetchware.

Just consider the description's of App::Fetchware's API below, and perhaps consult their full documentation in "FETCHWAREFILE API SUBROUTINES".

my $temp_dir = start(KeepTempDir = 0 | 1)> - Gives your extension a chance to do anything needed before the rest of the API subroutines get called. App::Fetchware's start() manages App::Fetchware's temporary directory creation. If you would like to also use a temporary directory, you can just use App::Fetchware::ExportAPI to "inherit" App::Fetchware's start() instead of implementing it yourself.
my $download_url = lookup() - Determines and returns a download url that download() receives and uses to download the archive for the program.o
my $package_path = download($tempd_dir, $download_url) - Downloads its provided $download_url argument.
verify($download_url, $package_path) - Verifies the integrity of your downloaded archive using gpg, sha1, or md5.
my $build_path = unarchive($package_path) - Unpacks the downloaded archive.
build($build_path) - Configures and compiles the downloaded archive.
install() - Installs the compiled archive.
end() - Cleans up the temporary directory that start() created. Can be overridden to do any other clean up tasks that your archive needs.
uninstall($build_path) - Uninstalls an already installed program installed with the same App::Fetchware extension.

Also, keep in mind the order in which these subroutines are called, what arguments they receive, and what their expected return value is.

install - start(), lookup(), download(), verify(), unarchive(), build(), install(), and end().
upgrade - Exactly the same as install.
uninstall - You might think its just uninstall(), but its not. uninstall() calls start(), download() (to copy the already installed fetchware package from the fetchware package database to the temporary directory), unarchive(), uninstall(), and end().

Use the above overview of App::Fetchware's API to design what each API subroutine keeping in mind its arguments and what its supposed to return.

Determine your fetchware extension's Fetchwarefile configuration options.

App::Fetchware has various configuration options such as temp_dir, prefix, and so on. Chances are your fetchware extension will also need such configuration options. These are easily created with App::Fetchware::CreateConfigOptions, which manufactures these to order for your convenience. There are four different types of configuration options:

ONE - Takes only one argument, and can only be used once.
ONEARRREF - Can only be called once, but can take multiple agruments at once.
MANY - Takes only one argument, but can be called more than once. The only example is mirror.
BOOLEAN - Takes only one arguement, and can only be called once just like ONE. The difference is that BOOLEANs are limited to only boolean true or false values such as 'On' or 'Off', 'True' or 'False', or 1 or 0. App::Fetchware examples include no_install and vefify_failure_ok.

Using the documentation above and perhaps also the documentation for App::Fetchware::CreateConfigOptions, determine the names of your configuration options, and what type of configuraton options they will be.

Implement your fetchware extension.

Since you've designed your new fetchware extension, now it's time to code it up. The easiest way to do so, is to just take an existing extension, and just copy and paste it, and then delete its specifics to create a simple extension skeleton. Then just follow the steps below to fill in this skeleton with the specifics needed for you fetchware extension.

1. Set up proper exports and imports.

Because fetchware needs your fetchware extension to export all of the subroutines that make up the fetchware's API, and any configuration options (as Perl subroutines) your extension will use, fetchware uses the helper packages App::Fetchware::ExportAPI and App::Fetchware::CreateConfigOptions to easily manage setting all of this up for you.

First, use App::Fetchware::ExportAPI to be sure to export all of fetchware's API subroutines. This package is also capable of "inheriting" any of App::Fetchware's API subroutines that you would like to keep. An example.

    # Use App::Fetchware::ExportAPI to set up proper exports this fetchware
    # extension.
    use App::Fetchware::ExportAPI KEEP => [qw(start end)],
        OVERRIDE => [qw(lookup download verify unarchive build install uninstall)]
    ;

Second, use App::Fetchware::CreateConfigOptions to create all of the configuration options (such as temp_dir, no_install, and so on.) you want your fetchware extension to have.

There are four types of configuration options.

ONE - Take one an only ever one argument, and can only be called once per Fetchwarefile.

Examples: temp_file, prefix, and lookup_url.

ONEARRREF - Takes one or more arguments like MANY, but unlike MANY can only be called once.

Examples: configure_options, make_options, and build_commands.

MANY - Takes one or more arguments, and can be called more than once. If called more than once, then second call's arguments are added to the existing list of arguments.

Examples: mirror.

BOOLEAN - Just like ONE except it will convert /off/i and /false/i to 0 to support more than just Perl's 0 or undef being false.

Examples: verify_failure_ok, no_install, and stay_root.

An example.

    use App::Fetchware::CreateConfigOptions
        IMPORT => [qw(temp_dir no_install)],
        ONE => [qw(repository directory)],
        ONEARRREF => [qw(build_options install_options)],
        BOOLEAN => [qw(delete_after_download)],
    ;

These 2 simple use()'s are all it takes to set up proper exports for your fetchware extension.

2. Code any App::Fetchware API subroutines that you won't be reusing from App::Fetchware.

Use their API documentation from the section "FETCHWAREFILE API SUBROUTINES" to ensure that you use the correct subroutine names, arguments and return the correct value as well.

An example for overriding lookup() is below.

    =head2 lookup()

        my $download_url = lookup();

    # New lookup docs go here....

    =cut

    sub lookup {

        # New code for new lookup() goes here....

        # Return the required $download_url.
        return $download_url;
    }

Use Fetchware's Own Libraries to Save Developement Time.

Fetchware includes many libraries to save development time. These libraries are well tested by Fetchware's own test suite, so you too can use them to save development time in your own App::Fetchware extensions.

These libraries are:

App::Fetchware::Util

Houses various logging subroutines, downloading subroutines, security subroutines, and temporary directory managment subroutines that fetchware itself uses, and you can also make use of them in your own fetchware extensions.

  • Logging subroutines - msg(), vmsg(), and run_prog().
    • These subroutines support fetchware's command line options such as -v (--verbose) and -q (--quiet).
    • msg() should be used to print a message to the screen that should always be printed, while vmsg() should only be used to print messages to the screen when the -v (--verbose) command line option is turned on.
    • run_prog() is a system() wrapper that also supports -v and -q options, and should be used to run any external commands that your App::Fetchware extension needs to run.
  • Downloading subroutines - download_file() and download_dirlist().
    • download_file() should be used to download files. It supports the following schemes ftp://, http://, or file://. It simply downloads the file using Net::Ftp or HTTP::Tiny to the current working directory.
    • download_dirlist() should be used to download FTP, HTTP, or local directory listings. It is mostly just used by lookup() to determine if a new version is available based on the information it parses from the directory listing.
  • Security subroutines - safe_open() and drop_privs().
    • safe_open() opens the file and then runs a bunch of file and directory tests to ensure that only the user running fetchware or root can modify the file or any of that file's containing directories to help prevent Fetchwarefiles from perhaps being tampered by other users or programs.
    • drop_privs() forks and drops privileges. It's used by fetchware to drop privs to avoid downloading and compiling software as root. It most likely should not be called by App::Fetchware extensions, because fetchware already calls it for you, but it is there if you need it.
  • Temporary Directory subroutines - create_tempdir(), original_cwd(), and cleanup_tempdir().
    • create_tempdir() creates and chdir()'s into a temporary directory using File::Temp's tempdir() function. It also deals with creating a Fetchware semaphore file to keep fetchware clean from deleting any still needed temporary directories.
    • original_cwd() simply returns what fetchware's current working directory was before create_tempdir() created and chdir()'d into the temporary directory.
    • cleanup_tempdir() deals with closing the fetchware semaphore file.
Test::Fetchware

Test::Fetchware includes utility subroutines that fetchware itself uses to test itself, and they are shared with extension writers through this module.

  • eval_ok() - A poor man's Test::Exception in one simple subroutine. Why require every user to install a dependency only used for testing when one simple subroutine does the trick.
  • print_ok() - A poor man's Test::Output in one simple subroutine.
  • skip_all_unless_release_testing() - Does just what it's name says. If fetchware's internal release/author only Environment variables are set, only then will any Test::More subtests that call this subroutine skip the entire subtest. This is used to skip running tests that install real programs on the testing computer's system. Many of Fetchware's tests actually install a real program such as Apache, and I doubt any Fetchware user would like to have Apache installed and uninstalled a bagillion times when they install Fetchware. Use this subroutine in your own App::Fetchware extension's to keep that from happening.
  • make_clean() - Just run_prog('make', 'clean') in the current working directory just as its name suggests.
  • make_test_dist() - Use this subroutine or craft your own similar subroutine, if you need more flexibility, to test actually installing a program on user's systems that just happens to execute all of the proper installation commands, but supplies installation commands that don't actually install anything. It's used to test actually installing software without actually installing any software.
  • md5sum_file() - Used to provide a md5sum file for make_test_dist() create software packages in order to pass fetchware's verify checks.
  • verbose_on() - Make's all vmsg()'s actually print to the screen even if -v or --verbose was not actually provided on the command line. Used to aid debugging.
App::Fetchware::Config

App::Fetchware::Config stores and manages fetchware's parsed configuration file. parse_fetchwarefile() from fetchware does the actual parsing, but it stores the configuration file inside App::Fetchware::Config. Use the subroutines below to access any configuration file options that you create with App::Fetchware::CreateConfigOptions to customize your fetchware extension. Also feel free to reuse any names of App::Fetchware configuration subroutines such as temp_dir or lookup_url

config() - Sets and gets values from the currently parsed fetchware configuration file. If there is one argument, then it returns that configuration options value or undef if there is none.If there are more than one argument, then the first argument is what configuration option to use, and the rest of the arguments are what values to set that configuration option to.
config_iter() - returns a configuration iterator. that when kicked (called, like $config_iter->()) will return one value from the specifed configuration option. Can be kicked any number of times, but once the number of configuration values is exhausted the iterator will return undef.
config_replace() - config() is used to set configuration options, and once set they cannot be changed by config(). This is meant to catch and reduce errors. But sometimes, mostly in test suites, you need to change the value of a configuration option. That's what config_replace() is for.
config_delete() - deletes the specified configuration option. Mostly just used for testing.
__clear_CONFIG() - An internal only subroutine that should be only used when it is really really needed. It clears (deletes) the entire internal hash that the configuration options are stored in. It really should only be used during testing to clear App::Fetchware::Config's intenal state between tests.
debug_CONFIG() - prints App::Fetchware::Config's internal state directly to STDOUT. Meant for debugging only in your test suite.
App::Fetchware's OVERRIDE_* export tags.

App::Fetchware's main API subroutines, especially the crazy complicated ones such as lookup(), are created by calling and passing data among many component subroutines. This is done to make testing much much easier, and to allow App::Fetchware extensions to also use some or most of these component subroutines when they override a App::Fetchware API subroutine.

lookup()'s OVERRIDE_LOOKUP export tag.

This export tag is the largest, and perhaps the most important, because it implements fetchware's ability to determine if a new version of your software package is available. Its default is just a clever use of HTTP and FTP directory listings.

See the section "lookup() API REFERENCE" for more details on how to use these subroutines to determine if new versions of your software is available automatically.

download()'s OVERRIDE_DOWNLOAD export tag.

Only exports the subroutine determine_package_path(), which simply comcatenates a $tempdir with a $filename to return a properl $package_path, which unarchive later uses. This is mostly its own subroutine to better document how this is done, and to allow easier code reuse.

verify()'s OVERRIDE_VERIFY export tag.

Exports a family of subroutines to verify via MD5, SHA1, or GPG the integrity of your downloaded package. MD5 and SHA1 are supported for legacy reasons. All software packages should be GPG signed for much much much better security. GPG signatures when verified actually prove that the software package you downloaded is exactly what the author of that software package created, whereas MD5 and SHA1 sums just verify that you downloaded the bunch of bits in the same order that they are stored on the server.

digest_verify() an be used to add support for any other Digest::* modules that CPAN has a Digest based module for that correctly follow Digest's API.

unarchive()'s OVERRIDE_UNARCHIVE export tag.

Exports subroutines that will help you unarchive software packages in tar and zip format. The most important part to remember is to use list_files() to list the files in your archive, and pass that list to check_archive_files() to ensure that the archive will not overwrite any system files, and contains no absolute paths that could cause havok on your system. unarchive_package() does the actual unarchiving of software packages.

build()'s OVERRIDE_BUILD export tag.

Provides run_star_commands(), which is meant to execute common override commands that fetchware provides with the build_commands, install_commands, and uninstall_commands configuration file directives. These directives are of type ONEARRREF where they can only be called once, but you can supply a comma separated list of commands that fetchware will install instead of the standard commands default AutoTools commands (build() => ./configure, make; install() => make install; uninstall() => ./configure, make uninstall). See its documentation for more details.

install()'s OVERRIDE_INSTALL export tag.

install() only exports chdir_unless_already_at_path(), which is of limited use. install() also uses build()'s run_star_commands().

uninstall()'s OVERRDIE_UNINSTALL export tag.

uninstall() actually has no exports of its own, but it does make use of build() and install()'s exports.

Write your fetchware extension's documentation

Fill in all of the skeleton's missing POD to ensure that fetchware extension has enough documentation to make it easy for user's to use your fetchware extension. Be sure to document:

Write tests for your fetchware extension

Use perls veneralbe Test::More, and whatever other Perl TAP testing modules you need to be sure your fetchware extension works as expected.

"" in Test::Fetchware has a few testing subroutines that fetchware itself uses in its test suite that you may find helpful. These include:

"eval_ok()" in Test::Fetchware - A poor man's Test::Exception. Captures any exceptions that are thrown, and compares them to the provided exception text or regex.
"print_ok()" in Test::Fetchware - A poor man's Test::Output. Captures STDOUT, and compares it to the provided text.
"skip_all_unless_release_testing()" in Test::Fetchware - Fetchware is a package manager, but who wants software installed on their computer just to test it? This subroutine marks test files or subtests that should be skipped unless fetchware's extensive FETCHWARE_RELEASE_TESTING environement variables are set. This funtionality is described next.
"make_clean()" in Test::Fetchware - Just runs make clean in the current directory.
"make_test_dist()" in Test::Fetchware - Creates a temporary distribution that is used for testing. This temporary distribution contains a ./configure and a Makefile that create no files, but can still be executed in the standard AutoTools way.
"md5sum_file()" in Test::Fetchware - Just md5sum's a file so verify() can be tested.
"expected_filename_listing()" in Test::Fetchware - Returns a string of crazy Test::Deep subroutines to test filename listings. Not quite as useful as the rest, but may come in handy if you're only changing the front part of lookup().

Your tests should make use of fetchware's own FETHWARE_RELEASE_TESTING environment variable that controls with the help of skip_all_unless_release_testing() if and where software is actually installed. This is done, because everyone who installs fetchware or your fetchware extension is really gonna freak out if its test suite installs apache or ctags just to test its package manager functionality. To use it:

1. Set up an automated way of enabling FETCHWARE_RELEASE_TESTING.

Just paste the frt() bash shell function below. Translating this to your favorite shell should be pretty straight forward. Do not just copy and paste it. You'll need to customize the specific FETCHWARE_* environment variables to whatever mirrors you want to use or whatever actual programs you want to test with. And you'll have to point the local (file://) urls to directories that actually exist on your computer.

    # Sets FETCHWARE_RELEASE_TESTING env vars for fully testing fetchware.
    frt() {
        if [ -z "$FETCHWARE_RELEASE_TESTING" ]
        then
            echo -n 'Setting fetchware_release_testing environment variables...';
            export FETCHWARE_RELEASE_TESTING='***setting this will install software on your computer!!!!!!!***'
            export FETCHWARE_FTP_LOOKUP_URL='ftp://carroll.cac.psu.edu/pub/apache/httpd'
            export FETCHWARE_HTTP_LOOKUP_URL='http://www.apache.org/dist/httpd/'
            export FETCHWARE_FTP_MIRROR_URL='ftp://carroll.cac.psu.edu/pub/apache/httpd'
            export FETCHWARE_HTTP_MIRROR_URL='http://mirror.cc.columbia.edu/pub/software/apache//httpd/'
            export FETCHWARE_FTP_DOWNLOAD_URL='ftp://carroll.cac.psu.edu/pub/apache/httpd/httpd-2.2.26.tar.bz2'
            export FETCHWARE_HTTP_DOWNLOAD_URL='http://mirrors.ibiblio.org/apache//httpd/httpd-2.2.26.tar.bz2'
            export FETCHWARE_LOCAL_URL='file:///home/dly/software/httpd-2.2.22.tar.bz2'
            export FETCHWARE_LOCAL_ZIP_URL='file:///home/dly/software/ctags-zip/ctags58.zip'
            export FETCHWARE_LOCAL_BUILD_URL='/home/dly/software/ctags-5.8.tar.gz'
            export FETCHWARE_LOCAL_UPGRADE_URL='file:///home/dly/software/fetchware-upgrade'
            echo 'done.'
        else
            echo -n 'Deleting fetchware_release_testing environment variables...';
            unset FETCHWARE_RELEASE_TESTING
            unset FETCHWARE_FTP_LOOKUP_URL
            unset FETCHWARE_HTTP_LOOKUP_URL
            unset FETCHWARE_FTP_MIRROR_URL
            unset FETCHWARE_HTTP_MIRROR_URL
            unset FETCHWARE_FTP_DOWNLOAD_URL
            unset FETCHWARE_HTTP_DOWNLOAD_URL
            unset FETCHWARE_LOCAL_URL
            unset FETCHWARE_LOCAL_ZIP_URL
            unset FETCHWARE_LOCAL_BUILD_URL
            unset FETCHWARE_LOCAL_UPGRADE_URL
            echo 'done.'
        fi
    }

Just run frt with no args to turn FETCHWARE_RELEASE_TESTING on, and run it once more to turn it off. Don't forget to reload your shell's configuration with:

    $ . ~/.bashrc # Or whatever file you added it to is named.

Then inside your test suite just import skip_all_unless_release_testing() from Test::Fetchware:

    use Test::Fetchware ':TESTING';
2. Call skip_all_unless_release_testing() as needed

In each subtest where you actually install anything other than a test distribution made with make_test_dist(), begin that subtest with a call to skip_all_unless_release_testing(), which will skip the whole subtest if FETCHWARE_RELEASE_TESTING is not setup properly.

    subtest 'test install() for real' => sub {
        # Only test if during release testing.
        skip_all_unless_release_testing();
        
        # Tests that actually install non-trivial distributions such as ones
        # made by make_test_dist() go here.

        ....
    };

If you dislike subtests, or otherwise don't want to use them, then put all of the tests that actually install something into a SKIP block after the other tests that all users will run at the end of your test file.

Share it on CPAN

Fetchware has no Web site or any other place to share fetchware extensions. But fetchware is written in Perl, so fetchware can just use Perl's CPAN. To learn how to create modules and upload them to CPAN please see Perl's own documentation. perlnewmod shows how to create new Perl modules, and how to upload them to CPAN. See Module::Starter for a simple way to create a skeleton for a new Perl module, and dzil is beyond amazing, but has insane dependencies and a significant learning curve.

FETCHWAREFILE API SUBROUTINES ^

The subroutines below are Fetchwarefile's API subroutines or helper subroutines for App::Fetchware's API subroutines. If you want information on fetchware's configuration file syntax, then see the section "FETCHWAREFILE CONFIGURATION SYNTAX" for more information. For additional information on how to keep or override these API subroutines in a fetchware extension see the section "CREATING A FETCHWARE EXTENSION"

start()

    my $temp_dir = start();
Configuration subroutines used:
temp_dir

Creates a temp directory using File::Temp, and sets that directory up so that it will be deleted by File::Temp when fetchware closes.

Returns the $temp_file that start() creates, so everything else has access to the directory they should use for storing file operations.

EXTENSION OVERRIDE NOTES

start() calls App::Fetchware::Util's create_tempdir() subroutine that cleans up the temporary directory. If your fetchware extension overrides start() or end(), you must call create_tempdir() or name your temproary directories in a manner that fetchware clean won't find them, so something that does not start with fetchware-*.

If you fail to do this, and you use some other method to create temporary directories that begin with fetchware-*, then fetchware clean may delete your temporary directories out from under your feet. To fix this problem:

  • Use App::Fetchware::Util's create_tempdir() in your start() and cleanup_tempdir() in your end().
  • Or, be sure not to name your temprorary directory that you create and manage yourself to begin with fetchware-*, which is the glob pattern fetchware clean uses. I recommend using something like App-FetchwareX-NameOfExtension-$$-XXXXXXXXXXXXXX as the name you would use in your File::Temp::temdir $pattern, with $$ being the special perlvar for the curent processes id.
drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • start() is called in the parent with root privileges. This is done, so that when the parent calls cleanup_tempdir() in its end() call, cleanup_tempdir() still has a valid filehandle to the fetchware semaphore file, which is used to keep fetchware clean from deleting fetchware's temporary directories out from under if you run a fetchware clean while another process is running another fetchware comand at the same time.

lookup()

    my $download_path = lookup();
Configuration subroutines used:
lookup_url
lookup_method
filter
user_agent

Accesses lookup_url as a http/ftp directory listing, and uses lookup_method to determine what the newest version of the software is available. This is combined with the path given in lookup_url, and return as $download_path for use by download().

LIMITATIONS lookup_url is a web browser like URL such as http://host.com/a/b/c/path, and it must be a directory listing not a actual file. This directory listing must be a listing of all of the available versions of the program this Fetchwarefile belongs to.

Only ftp://, http://, and file:// URL scheme's are supported.

And the HTML directory listings Apache and other Web Server's return are exactly what lookup() uses to determine what the latest version available for download is.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • Under drop_privs() lookup() is executed in the child with reduced privileges.

lookup_method can be either 'timestamp' or 'versionstring', any other values will result in fetchware throwing an exception.

lookup() API REFERENCE

The subroutines below are used by lookup() to provide the lookup functionality for fetchware. If you have overridden the lookup() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from lookup.

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

check_lookup_config()

    check_lookup_config();

Verifies the configurations parameters lookup() uses are correct. These are lookup_url and lookup_method. If they are wrong die() is called. If they are right it does nothing, but return.

check_lookup_config() also currently checks if a program config option is given, and throws an exception if it is not.

It also ensures that you specify at least one mirror, and how to verify your program. If this is not done properly, an exception is thrown. For verification to work properly, you must specify gpg_keys_url, md5_url, or sha1_url. Only if there is no verification availabe from your program's author should you use the verification_failure 'On'; option.

get_directory_listing()

    my $directory_listing = get_directory_listing();

Downloads a directory listing that lookup() uses to determine what the latest version of your program is. It is returned.

SIDE EFFECTS

get_directory_listing() returns a SCALAR REF of the output of HTTP::Tiny or a ARRAY REF for Net::Ftp downloading that listing. Note: the output is different for each download type. Type 'http' will have HTML crap in it, and type 'ftp' will be the output of Net::Ftp's dir() method.

Type 'file' will just be a listing of files in the provided lookup_url directory.

parse_directory_listing()

    my $file_listing = parse_directory_listing($directory_listing);

Based on URL scheme of 'file', 'http', or 'ftp', parse_directory_listing() will call file_parse_filelist(), ftp_parse_filelist(), or http_parse_filelist(). Those subroutines do the heavy lifting, and the results are returned.

determine_download_path()

    my $download_path = determine_download_path($filename_listing);

Runs the lookup_method to determine what the lastest filename is, and that one is then concatenated with lookup_url to determine the $download_path, which is then returned to the caller.

Also calls lookup_determine_downloadpath() to determine the actual download path from the $sorted_filename_listing returned by lookup_by_*().

ftp_parse_filelist()

    $filename_listing = ftp_parse_filelist($ftp_listing);

Takes an array ref as its first parameter, and parses out the filenames and timstamps of what is assumed to be Net::FTP-dir()> long directory listing output.

Returns an array of arrays of filenames and timestamps.

http_parse_filelist()

    $filename_listing = http_parse_filelist($http_listing);

Takes an scalar of downloaded HTML output, and parses it using HTML::TreeBuilder to build and return an array of arrays of filenames and timestamps.

file_parse_filelist()

    my $filename_listing = file_parse_filelist($file_listing);

Parses the provided filelist by stating each file, and creating a properly formatted timestamp to return.

Returns an array of arrays of filenames and timestamps.

lookup_by_timestamp()

    my $download_url = lookup_by_timestamp($filename_listing);

Implements the 'timestamp' lookup algorithm. It takes the timestamps placed in its first argument, normalizes them into a standard descending format (YYYYMMDDHHMM), and then cleverly uses sort to determine the latest filename, which should be the latest version of your program.

lookup_by_versionstring()

    my $sorted_filename_listing = lookup_by_versionstring($filename_listing);

Determines the $sorted_filename_listing used by determine_download_path() by cleverly spliting the filenames (the first value of the array of arrays input, $filename_listing) on /\D+/, which will return a list of version numbers. Then, the cleverly splitted filename is pushed on to the input array, and then the array is sorted based on this new value using a custom sort block.

lookup_by_versionstring() also discards any entries from $filename_listing that do not have a version number in them, because without version numbers that entry can not be sorted properly. And if it is left in, it could confuse fetchware into figuring out the correct download path.

Also note, that both $filename_listing and lookup_by_versionstring()'s return value, $sorted_filename_listing, must both be arrayrefs of arrays. That is a scallar pointing to an array of arrays.

lookup_determine_downloadpath()

    my $download_path = lookup_determine_downloadpath($file_listing);

Given a $file_listing of files with the same timestamp or versionstring, determine which one is a downloadable archive, a tarball or zip file. And support some hacks to make fetchware more robust. These are the filter configuration subroutine, ignoring "win32" on non-Windows systems, and supporting Apache's CURRENT_IS_ver_num and Linux's LATEST_IS_ver_num helper files.

download()

    my $package_path = download($temp_dir, $download_path);
Configuration subroutines used:
mirror
user_agent

Downloads $download_path to tempdir 'whatever/you/specify'; or to whatever File::Spec's tempdir() method tries. Supports ftp and http URLs as well as local files specified like in browsers using file://

Also, returns $package_path, which is used by unarchive() as the path to the archive for unarchive() to untar or unzip.

LIMITATIONS Uses Net::FTP and HTTP::Tiny to download ftp and http files. No other types of downloading are supported, and fetchware is stuck with whatever limitations or bugs Net::FTP or HTTP::Tiny impose.
drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • Under drop_privs() download() is executed in the child with reduced privileges.

download() API REFERENCE

The subroutines below are used by download() to provide the download functionality for fetchware. If you have overridden the download() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from download.

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

determine_package_path()

    my $package_path = determine_package_path($tempdir, $filename)

Determines what $package_path is based on the provided $tempdir and $filename. $package_path is the path used by unarchive() to unarchive the software distribution download() downloads.

$package_path is returned to caller.

verify()

    verify($download_path, $package_path)
Configuration subroutines used:
gpg_keys_url
user_keyring
gpg_sig_url
sha1_url
md5_url
verify_method
verify_failure_ok
user_agent

Verifies the downloaded package stored in $package_path by downloading $download_path.{asc,sha1,md5}> and comparing the two together. Uses the helper subroutines {gpg,sha1,md5,digest}_verify().

LIMITATIONS Uses gpg command line, and the interface to gpg is a little brittle. Crypt::OpenPGP is buggy and not currently maintainted again, so fetchware cannot make use of it, so were stuck with using the command line gpg program.
drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • Under drop_privs() verify() is executed in the child with reduced privileges.

verify() API REFERENCE

The subroutines below are used by verify() to provide the verify functionality for fetchware. If you have overridden the verify() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from verify().

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

gpg_verify()

    'Package Verified' = gpg_verify($download_path);

Uses the command-line program gpg to cryptographically verify that the file you download is the same as the file the author uploaded. It uses public-key priviate-key cryptography. The author signs his software package using gpg or some other OpenPGP compliant program creating a digital signature file with the same filename as the software package, but usually with a .asc file name extension. gpg_verify() downloads the author's keys, imports them into fetchware's own keyring unless the user sets user_keyring to true in his Fetchwarefile. Then Fetchware downloads a digital signature that usually ends in .asc. Afterwards, fetchware uses the gpg command line program to verify the digital signature. gpg_verify returns true if successful, and throws an exception otherwise.

You can use gpg_keys_url to specify the URL of a file where the author has uploaded his keys. And the gpg_sig_url can be used to setup an alternative location of where the .asc digital signature is stored.

sha1_verify()

    'Package verified' = sha1_verify($download_path, $package_path);
    undef = sha1_verify($download_path, $package_path);

Verifies the downloaded software archive's integrity using the SHA Digest specified by the sha_url 'ftp://sha.url/package.sha' config option. Returns true for sucess dies on error.

SECURITY NOTE If an attacker cracks a mirror and modifies a software package, they can also modify the MD5 sum of that software package on that same mirror. Because of this limitation MD5 sums can only tell you if the software package was corrupted while downloading. This can actually happen as I've had it happen to me once.

If your stuck with using MD5 sum, because your software package does not provide gpg signing, I recommend that you download your SHA1 sums (and MD5 sums) from your software package's master mirror. For example, Apache provides MD5 and SHA1 sums, but it does not mirror them--you must download them directly from Apache's servers. To do this specify a sha1_url 'master.mirror/package.sha1'; in your Fetchwarefile.

md5_verify()

    'Package verified' = md5_verify($download_path, $package_path);
    undef = md5_verify($download_path, $package_path);

Verifies the downloaded software archive's integrity using the MD5 Digest specified by the md5_url 'ftp://sha.url/package.sha' config option. Returns true for sucess and dies on error.

SECURITY NOTE If an attacker cracks a mirror and modifies a software package, they can also modify the MD5 sum of that software package on that same mirror. Because of this limitation MD5 sums can only tell you if the software package was corrupted while downloading. This can actually happen as I've had it happen to me once.

If your stuck with using MD5 sum, because your software package does not provide gpg signing, I recommend that you download your MD5 sums (and SHA1 sums) from your software package's master mirror. For example, Apache provides MD5 and SHA1 sums, but it does not mirror them--you must download them directly from Apache's servers. To do this specify a md5_url 'master.mirror/package.md5'; in your Fetchwarefile.

digest_verify()

    'Package verified' = digest_verify($digest_type, $download_path, $package_path);
    undef = digest_verify($digest_type, $download_path, $package_path);

Verifies the downloaded software archive's integrity using the specified $digest_type, which also determines the "$digest_type_url" 'ftp://sha.url/package.sha' config option. Returns true for sucess and returns false for failure.

OVERRIDE NOTE If you need to override verify() in your Fetchwarefile to change the type of digest used, you can do this easily, because digest_verify() uses Digest, which supports a number of Digest::* modules of different Digest algorithms. Simply do this by override verify() to call digest_verify('Digest's name for your Digest::* algorithm');
SECURITY NOTE If an attacker cracks a mirror and modifies a software package, they can also modify the $digest_type sum of that software package on that same mirror. Because of this limitation $digest_type sums can only tell you if the software package was corrupted while downloading. This can actually happen as I've had it happen to me once.

If your stuck with using $digest_type sum, because your software package does not provide gpg signing, I recommend that you download your $digest_type sums (and SHA1 sums) from your software package's master mirror. For example, Apache provides MD5 and SHA1 sums, but it does not mirror them--you must download them directly from Apache's servers. To do this specify a $digest_type_url 'master.mirror/package.$digest_type';' in your Fetchwarefile.

unarchive()

    my $build_path = unarchive($package_path)
Configuration subroutines used:
none

Uses Archive::Tar or Archive::Zip to turn .tar.{gz,bz2,xz} or .zip into a directory. Is intelligent enough to warn if the archive being unarchived does not contain all of its files in a single directory like nearly all software packages do. Uses $package_path as the archive to unarchive, and returns $build_path.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • Under drop_privs() unarchive() is executed in the child with reduced privileges.

unarchive() API REFERENCE

The subroutine below are used by unarchive() to provide the unarchive functionality for fetchware. If you have overridden the unarchive() handler, you may want to use some of these subroutines so that you don't have to copy and paste anything from unarchive().

App::Fetchware is not object-oriented; therefore, you can not subclass App::Fetchware to extend it!

list_files()

    # Remember $files is a array ref not a regular old scalar.
    my $files = list_files($package_path;

list_files() takes $package_path as an argument to a tar'ed and compressed package or to a zip package, and calls list_files_{tar,zip}() accordingly. list_files_{tar,zip}() in turn uses either Archive::Tar or Archive::Zip to list all of the files in the archive and return a arrayref to that list.

list_files_tar()

    my $tar_file_listing = list_files_tar($path_to_tar_archive);

Returns a arrayref of file names that are found in the given, $path_to_tar_archive, tar file. Throws an exception if there is an error.

It uses Archive::Tar->iter() to avoid reading the entire tar archive into memory.

list_files_zip()

    my $zip_file_listing = list_files_zip($path_to_zip_archive);

Returns a arrayref of file names that are found in the given, $path_to_zip_archive, zip file. Throws an exception if there is an error.

unarchive_package()

    unarchive_package($format, $package_path)

unarchive_package() unarchive's the $package_path based on the type of $format the package is. $format is determined and returned by list_archive(). The currently supported types are tar and zip. Nothing is returned, but unarchive_{tar,zip}() is used to to use Archive::Tar or Archive::Zip to unarchive the specified $package_path.

unarchive_tar()

    my @extracted_files = unarchive_tar($path_to_tar_archive);

Extracts the given $path_to_tar_archive. It must be a tar archive. Use unarchive_zip() for zip archives. It returns a list of files that it extracted.

unarchive_zip()

    'Extraced files successfully.' = unarchive_zip($path_to_zip_archive);

Extraces the give $path_to_zip_archive. It must be a zip archive. Use unarchive_tar() for tar archives. It only returns true for success. It does not return a list of extracted files like unarchive_tar() does, because Archive::Zip's extractTree() method does not.

check_archive_files()

    my $build_path = check_archive_files($files);

Checks if all of the files in the archive are contained in one main directory, and spits out a warning if they are not. Also checks if one or more of the files is an absolute path, and if so it throws an exception, because absolute paths could potentially overwrite important system files messing up your computer.

build()

    'build succeeded' = build($build_path)
Configuration subroutines used:
prefix
configure_options
make_options
build_commands

Changes directory to $build_path, and then executes the default build commands of './configure', 'make', 'make install' unless your Fetchwarefile specifies some build options like build_commands 'exact, build, commands';, make_options '-j4';, configure_options '--prefix=/usr/local';, or prefix '/usr/local'.

You can only specify build_commands or any of the other three build options. You cannot combine build_commands with any of the other build options.

LIMITATIONS build() like install() inteligently parses build_commands, prefix, make_options, and configure_options by just using Test::Parsewords to parse out the string considering quotes, and then execute them using fetchware's built in run_prog() to properly support -q.

Also, simply executes the commands you specify or the default ones if you specify none. Fetchware will check if the commands you specify exist and are executable, but the kernel will do it for you and any errors will be in fetchware's output.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • Under drop_privs() build() is executed in the child with reduced privileges.

    The $build_path it returns is very important, because the parent process will need to know this path, which is in turn provided to install() as an argument, and install will then chdir to $build_path.

build() API REFERENCE

Below are the subroutines that build() exports with its BUILD_OVERRIDE export tag. run_configure() runs ./configure, and is needed, because AutoTools uses full paths in its configuration files, so if you move it to a different location or a different machine, the paths won't match up, which will cause build and install errors. Rerunning ./configure with the same options as before will recreate the paths to get rid of the errors.

run_star_commands()

    run_star_commands(config('*_commands'));

run_star_commands() exists to remove some crazy copy and pasting from build(), install(), and uninstall(). They all loop over the list accessing a ONEARRREF configuration option such as config('build_commands'), and then determine what the individual star_commands are, and then run them with run_prog().

The "star" simply refers to shell globbing with the "star" character * meaning "any."

run_configure()

    run_configure();

Runs ./configure as part of build() or uninstall(), which also annoying needs to run it.

NOTE run_configure() is a piece of build() that was chopped out, because uninstall() needs to run ./configure too. The only reason uninstall() must do this is because Autotools uses full paths in the Makefile it creates. Examples from Apache are pasted below.
    top_srcdir   = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
    top_builddir = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
    srcdir       = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
    builddir     = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22
    VPATH        = /tmp/fetchware-5506-8ROnNQObhd/httpd-2.2.22

Why arn't relative paths good enough for Autotools?

install()

    'install succeeded' = install($build_path);
Configuration subroutines used:
install_commands
make_options
prefix

Executes make install, which installs the specified software, or executes whatever install_commands 'install, commands'; if its defined.

install() takes $build_path as its argument, because it must chdir() to this path if fetchware drops privileges.

LIMITATIONS

install() like build() inteligently parses install_commands by split()ing on /,\s+/, and then split()ing again on ' ', and then execute them fetchware's built-in run_prog(), which supports -q.

Also, simply executes the commands you specify or the default ones if you specify none. Fetchware will check if the commands you specify exist and are executable, but the kernel will do it for you and any errors will be in fetchware's output.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • install() is run in the parent process as root, because most server programs must be installed as root. install() must be called with $build_path as its argument, because it may need to chdir to $build_path, the same path that build() built the package in, in order to be able to install the program.

install() API REFERENCE

Below are the subroutines that install() exports with its INSTALL_OVERRIDE export tag. fillmein!!!!!!!!

chdir_unless_already_at_path()

chdir_unless_already_at_path() takes a $path as its argument, and determines if that path is currently part of the current processes current working directory. If it is, then it does nothing. Buf if the given $path is not in the current working directory, then it is chdir()'d to.

If the chdir() fails, an exception is thrown.

uninstall()

    'uninstall succeeded' = uninstall($build_path)
Configuration subroutines used:
uninstall_commands

Cd's to $build_path, and then executes make uninstall, which installs the specified software, or executes whatever uninstall_commands 'uninstall, commands'; if its defined.

LIMITATIONS

uninstall() like install() inteligently parses install_commands by split()ing on /,\s+/, and then split()ing again on ' ', and then execute them fetchware's built-in run_prog(), which supports -q.

Also, simply executes the commands you specify or the default ones if you specify none. Fetchware will check if the commands you specify exist and are executable, but the kernel will do it for you and any errors will be in fetchware's output.

drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • uninstall() is run in the parent process as root, because most server programs must be uninstalled as root. uninstall() must be called with $build_path as its argument, because it may need to chdir to $build_path, the same path that build() built the package in, in order to be able to uninstall the program.

end()

    end();
Configuration subroutines used:
none

end() is called after all of the other main fetchware subroutines such as lookup() are called. It's job is to cleanup after everything else. It just calls cleanup_tempdir(), which mostly just closes the fetchware.sem fetchware semaphore file used to lock each fetchware temporary directory so fetchware clean does not delete it.

It also calls the very internal only __clear_CONFIG() subroutine that clears App::Fetchware's internal %CONFIG variable used to hold your parsed Fetchwarefile.

EXTENSION OVERRIDE NOTES

end() calls App::Fetchware::Util's cleanup_tempdir() subroutine that cleans up the temporary directory. If your fetchware extension overrides end() or start(), you must call cleanup_tempdir() or name your temproary directories in a manner that fetchware clean won't find them, so something that does not start with fetchware-*.

If you fail to do this, and you use some other method to create temporary directories that begin with fetchware-*, then fetchware clean may delete your temporary directories out from under your feet. To fix this problem:

  • Use App::Fetchware::Util's create_tempdir() in your start() and cleanup_tempdir() in your end().
  • Or, be sure not to name your temprorary directory that you create and manage yourself to begin with fetchware-*, which is the glob pattern fetchware clean uses.
drop_privs() NOTES

This section notes whatever problems you might come accross implementing and debugging your Fetchware extension due to fetchware's drop_privs mechanism.

See Util's drop_privs() subroutine for more info.

  • end() runs in the parent process similar to start(). It needs to be able to close the lockfile that start() created so that fetchware clean cannot delete fetchware's temporary directory out from under it.

EXAMPLE FETCHWAREFILES ^

Below are example Fetchwarefiles. They use a range of features to show you what Fetchware can do, and how it's Fetchwarefile can be manipulated to work with any source-code distribution.

Apache Web Server

This Apache Fetchwarefile includes a few extra mirrors just in case one is down. The fairly common -j 4 make_option to make the build go faster, and a gigantic configure_options telling ./configure excatly how I want my Apache built and configured. It also uses a heredoc to make its configure_options configuration option much more legible.

    use App::Fetchware;

    program 'Apache';
    lookup_url 'http://www.apache.org/dist/httpd/';
    filter 'httpd-2.2';
    mirror 'http://apache.mirrors.pair.com/httpd/';
    mirror 'http://mirrors.ibiblio.org/apache/httpd/';
    mirror 'ftp://apache.cs.utah.edu/apache.org/httpd/';

    verify_method 'gpg';
    gpg_keys_url 'http://www.apache.org/dist/httpd/KEYS';

    make_options '-j 4';
    prefix '/home/dly/software/apache2.2';
    # You can use heredocs to make gigantic options like this one more legible.
    configure_options <<EOO;
    --with-mpm=prefork
    --enable-modules="access alias auth autoindex cgi logio log_config status vhost_alias userdir rewrite ssl"
    --enable-so
    EOO

NGINX Web Server

nginx has its distribution set up differently than Apache, so some changes are needed. First, nginx does not seem to use any mirrors at all, which means nginx's Fetchwarefile is going to look kind of stupid with the same exact URL being used for both the lookup_url and the mirror, but such a configuration is supported. Next, nginx does not have a KEYS file, but it does list it's developer's keys on its Website. So, they have to be imported manually into your keyring, and then specify the user_keyring option to switch fetchware from usings its own keyring to using your own keyring. Also, note the comment regarding having to use the user option to specify a real user account. This is needed, because the verify step is done by fetchware's child after that child drops its root privileges. Th default user is nobody, and nobody has no real home, and therefore no keyring, so gpg won't be able to read the keys you ask it to by using the user_keyring option; therefore, user must be specified to change it to a real user, whoose keyring has had these keys imported into it. Also, worth noting that this nginx configuration does not use a filter option. This is not actually needed, because the only source-code packages availabe at the lookup_url are the nginx software packages themselves, but it might be a good idea to include one, because the nginx developers could always change how their download server is structured. So, including it is always a good idea.

    use App::Fetchware;
    
    program 'nginx';
    
    # lookup_url and mirror are the same thing, because nginx does not seem to have
    # mirrors. Fetchware, however, requires one, so the same URL is simply
    # duplicated.
    lookup_url 'http://nginx.org/download/';
    mirror 'http://nginx.org/download/';
    
    
    # Must add the developers public keys to my own keyring. These keys are
    # availabe from http://nginx.org/en/pgp_keys.html Do this with:
    # gpg \
    # --fetch-keys http://nginx.org/keys/aalexeev.key\
    # --fetch-keys http://nginx.org/keys/is.key\
    # --fetch-keys http://nginx.org/keys/mdounin.key\
    # --fetch-keys http://nginx.org/keys/maxim.key\
    # --fetch-keys http://nginx.org/keys/sb.key\
    # --fetch-keys http://nginx.org/keys/glebius.key\
    # --fetch-keys http://nginx.org/keys/nginx_signing.key
    # You might think you could just set gpg_keys_url to the nginx-signing.key key,
    # but that won't work, because like apache different releases are signed by
    # different people. Perhaps I could change gpg_keys_url to be like mirror where
    # you can specify more than one option?
    user_keyring 'On';
    # user_keyring specifies to use the user's own keyring instead of fetchware's.
    # But fetchware drops privileges by default using he user 'nobody.' nobody is
    # nobody, so that user account does not have a home directory for gpg to read a
    # keyring from. Therefore, I'm using my own account instead.
    user 'dly';
    # The other option, which is commented out below, is to use root's own keyring,
    # and the no_install option to ensure that root uses its own keyring instead of
    # nobody's.
    # noinstall 'On';
    verify_method 'gpg';

PHP Programming Language

PHP annoyingly uses a custom Web application on each of its mirror sites to serve HTTP downloads. No simple directory listing is available. Therefore, to use php with fetchware, custom lookup, download, and verify hooks are needed that override fetchware's internal behavior to customize fetchware as needed so that it can work with how PHP's site is up.

The lookup hook downloads and parses the http://www.php.net/downloads.php page, which lists files availabe for download. This file is parsed using HTML::TreeBuilder to determine the latest version. The MD5 sum is also parsed out to verify the downloaded file as well.

The download hook is only needed, because http_download_file() presumes that the last part of the path is the filename you're downloading. And this is annoyingly not the case with the way PHP has its downloading system set up.

The verify hook just uses Digest::MD5 to calculate the md5sum of the downloaded file, and compares it with the one lookup parses out.

    use App::Fetchware qw(
        :OVERRIDE_LOOKUP
        :OVERRIDE_DOWNLOAD
        :OVERRIDE_VERIFY
        :DEFAULT
    );
    use App::Fetchware::Util ':UTIL';
    use HTML::TreeBuilder;
    use URI::Split qw(uri_split uri_join);
    use Data::Dumper;
    use HTTP::Tiny;
    
    program 'php';
    
    lookup_url 'http://us1.php.net/downloads.php';
    mirror 'http://us1.php.net';
    mirror 'http://us2.php.net';
    mirror 'http://www.php.net';
    
    # php does *not* use a standard http or ftp mirrors for downloads. Instead, it
    # uses its Web site, and some sort of application to download files using URLs
    # such as: http://us1.php.net/get/php-5.5.3.tar.bz2/from/this/mirror
    #
    # Bizarrely a URL like
    # http://us1.php.net/get/php-5.5.3.tar.bz2/from/us2.php.net/mirror
    # gets you the same page, but on a different mirror. Weirdly, these are direct
    # downloads without any HTTP redirects using 300 codes, but direct downloads.
    # 
    # This is why using fetchware with php you needs a custom lookup handler.
    # The files you download are resolved to a [http://us1.php.net/distributions/...]
    # directory, but trying to access a apache styple auto index at that url fails
    # with a rediret back to downloads.php.
    my $md5sum;
    hook lookup => sub {
        die <<EOD unless config('lookup_url') =~ m!^http://!;
    php.Fetchwarefile: Only http:// lookup_url's and mirrors are supported. Please
    only specify a http lookup_url or mirror.
    EOD
    
        msg "Downloading lookup_url [@{[config('lookup_url')]}].";
        my $dir_list = download_dirlist(config('lookup_url'));
    
        vmsg "Parsing HTML page listing php releases.";
        my $tree = HTML::TreeBuilder->new_from_content($dir_list);
    
        # This parsing code assumes that the latest version of php is the first one
        # we find, which seems like a dependency that's unlikely to change.
        my $download_path;
        $tree->look_down(
            _tag => 'a',
            sub {
                my $h = shift;
                
                my $link = $h->as_text();
    
                # Is the link a php download link or something to ignore.
                if ($link =~ /tar\.(gz|bz2|xz)|(tgz|tbz2|txz)/) {
    
                    # Set $download_path to this tags href, which should be
                    # something like: /get/php-5.5.3.tar.bz2/from/a/mirror
                    if (exists $h->{href} and defined $h->{href}) {
                        $download_path = $h->{href};
                    } else {
                        die <<EOD;
    php.Fetchwarefile: A path should be found in this link [$link], but there is no
    path it in. No href [$h->{href}].
    EOD
                    }
    
                    # Find and save the $md5sum for the verify hook below.
                    # It should be 3 elements over, so it should be the third index
                    # in the @right array below (remember to start counting 2 0.).
                    my @right = $h->right();
                    my $md5_span_tag = $right[2];
                    $md5sum = $md5_span_tag->as_text();
                    $md5sum =~ s/md5:\s+//; # Ditch md5 header.
                }
            }
        );
    
        # Delete the $tree, so perl can garbage collect it.
        $tree = $tree->delete;
    
        # Determine and return a proper $download_path.
        # Switch it from [/from/a/mirror] to [/from/this/mirror], so the mirror will
        # actually return the file to download.
        $download_path =~ s!/a/!/this/!;
    
        vmsg "Determined download path to be [$download_path]";
        return $download_path;
    };
    
    
    # I also must hook download(), because fetchware presumes that the filename of
    # the downloaded file is the last part of the $path, but that is not the case
    # with the path php uses for file downloads, because it ends in mirror, which is
    # *not* the name of the file; therefore, I must  hook download() to fix this
    # problem.
    hook download => sub {
        my ($temp_dir, $download_path) = @_;
    
        my $http = HTTP::Tiny->new();
        my $response;
        for my $mirror (config('mirror')) {
            my ($scheme, $auth, $path, $query, $fragment) = uri_split($mirror);
            my $url = uri_join($scheme, $auth, $download_path, undef, undef);
            msg <<EOM;
    Downloading path [$download_path] using mirror [$mirror].
    EOM
            $response = $http->get($url);
            
            # Only download it once.
            last if $response->{success};
        }
    
        die <<EOD unless $response->{success};
    php.Fetchwarefile: Failed to download the download path [$download_path] using
    the mirrors [@{[config('mirror')]}]. The response was:
    [@{[Dumper($response->{headers})]}].
    EOD
        die <<EOD unless length $response->{content};
    php.Fetchwarefile: Didn't actually download anything. The length of what was
    downloaded is zero. status [$response->{status}] reason [$response->{reason}]
    HTTP headers [@{[Dumper($response->{headers})]}].
    EOD
    
        msg 'File downloaded successfully.';
    
        # Determine $filename from $download_path
        my @paths = split('/', $download_path);
        my ($filename) = grep /php/, @paths;
    
        vmsg "Filename determined to be [$filename]";
    
        open(my $fh, '>', $filename) or die <<EOD;
    php.Fetchwarefile: Failed to open [$filename] for writing. OS error [$!].
    EOD
    
        print $fh $response->{content};
        close $fh or die <<EOD;
    php.Fetchwarefile: Huh close($filename) failed! OS error [$!].
    EOD
    
        my $package_path = determine_package_path($temp_dir, $filename);
    
        vmsg "Package path determined to be [$package_path].";
    
        return $package_path
    };
    
    
    # The above lookup hook parses out the md5sum on the php downloads.php web
    # site, and stores it in $md5sum, which is used in the the verify hook below.
    hook verify => sub {
        # Don't need the $download_path, because lookup above did that work for us.
        # $package_path is the actual php file that we need to ensure its md5
        # matches the one lookup determined.
        my ($download_path, $package_path) = @_;
    
        msg "Verifying [$package_path] using md5.";
    
        dir <<EOD if not defined $md5sum;
    php.Fetchwarefile: lookup failed to figure out the md5sum for verify to use to
    verify that the php version [$package_path] matches the proper md5sum.
    The md5sum was [$md5sum].
    EOD
    
        my $package_fh = safe_open($package_path, <<EOD);
    php.Fetchwarefile: Can not open the php package [$package_path]. The OS error
    was [$!].
    EOD
    
        # Calculate the downloaded php file's md5sum.
        my $digest = Digest::MD5->new();
        $digest->addfile($package_fh);
        my $calculated_digest = $digest->hexdigest();
    
        die <<EOD unless $md5sum eq $calculated_digest;
    php.Fetchwarefile: MD5sum comparison failed. The calculated md5sum
    [$calculated_digest] does not match the one parsed of php.net's Web site
    [$md5sum]! Do not trust this downloaded file! Perhaps there's a bug somewhere,
    or perhaps the php mirror you downloaded this php package from has been hacked.
    Mirrors do get hacked occasionally, so it is very much possible.
    EOD
    
        msg "ms5sums [$md5sum] [$calculated_digest] match.";
    
        return 'Package Verified';
    };

PHP Programming Language using its git VCS instead of download mirrors.

PHP like most open source software you can easily download off the internet uses a version control system to track changes to its source code. This source code repository is basically the same thing as a normal source code distribution would be except VCS commands like git pull are used to update it instead of checking a mirror for a new version. The Fetchwarefile below for php customizes Fetchware to work with php's VCS instead of the traditional downloading of actual source code archives.

It overrides lookup() to use a local git repo stored in the $git_repo_dir variable. To create a repo just clone php's git repo (see http://us1.php.net/git.php for details.). It runs git pull to update the repo, and then it runs git tags, and ditches some older junk tags, and finds only the tags used for new versions of php. These are sorted using the versonstring lookup() algorithm, and the latest one is returned.

download() uses git checkout [latesttag] to "download" php by simply changing the working directory to the latest tag. verify() uses git's cool verify-tag command to verify the gpg signature. unarchive() is updated to do nothing since there is no archive to unarchive. However, because we reuse build(), archive() must return a $build_path that build() will change its directory to. start() and end() are also overridden, because managing a temporary directory is not needed, so, instead, they just do a git checkout master to switch from whatever the latest tag is back to master, because git pull bases what it does on what branch you're in, so we must actually be a real branch to update git.

    # php-using-git.Fetchwarefile: example fetchwarefile using php's git repo
    # for lookup(), download(), and verify() functionality.
    use App::Fetchware qw(:DEFAULT :OVERRIDE_LOOKUP);
    use App::Fetchware::Util ':UTIL';
    use Cwd 'cwd';
    
    # The directory where the php source code's local git repo is.
    my $git_repo_dir = '/home/dly/Desktop/Code/php-src';
    
    # By default Fetchware drops privs, and since the source code repo is stored in
    # the user dly's home directory, I should drop privs to dly, so that I have
    # permission to access it.
    user 'dly';
    
    # Determine latest version by using the tags developers create to determine the
    # latest version.
    hook lookup => sub {
        # chdir to git repo.
        chdir $git_repo_dir or die <<EOD;
    php.Fetchwarefile: Failed to chdir to git repo at
    [$git_repo_dir].
    OS error [$!].
    EOD
    
        # Pull latest changes from php git repo.
        run_prog('git pull');
    
        # First determine latest version that is *not* a development version.
        # And chomp off their newlines.
        chomp(my @tags = `git tag`);
    
        # Now sort @tags for only ones that begin with 'php-'.
        @tags = grep /^php-/, @tags;
    
        # Ditch release canidates (RC, alphas and betas.
        @tags = grep { $_ !~ /(RC\d+|beta\d+|alpha\d+)$/ } @tags;
    
        # Sort the tags to find the latest one.
        # This is quite brittle, but it works nicely.
        @tags = sort { $b cmp $a } @tags;
    
        # Return $download_path, which is only just the latest tag, because that's
        # all I need to know to download it using git by checking out the tag.
        my $download_path = $tags[0];
    
        return $download_path;
    };
    
    
    # Just checkout the latest tag to "download" it.
    hook download => sub {
        my ($temp_dir, $download_path) = @_;
    
        # The latest tag is the download path see lookup.
        my $latest_tag = $download_path;
    
        # checkout the $latest_tag to download it.
        run_prog('git checkout', "$latest_tag");
    
        my $package_path = cwd();
        return $package_path;
    };
    
    
    # You must manually add php's developer's gpg keys to your gpg keyring. Do
    # this by  going to the page: http://us1.php.net/downloads.php . At the
    # bottom the gpg key "names are listed such as "7267B52D" or "5DA04B5D."
    # These are their key "names." Use gpg to download them and import them into
    # your keyring using: gpg --keyserver pgp.mit.edu --recv-keys [key id]
    hook verify => sub {
        my ($download_path, $package_path) = @_;
    
        # the latest tag is the download path see lookup.
        my $latest_tag = $download_path;
    
        # Run git verify-tag to verify the latest tag
        my $success = eval { run_prog('git verify-tag', "$latest_tag"); 1;};
    
        # If the git verify-tag fails, *and* verify_failure_ok has been turned on,
        # then ignore the thrown exception, but print an annoying message.
        unless (defined $success and $success) {
            unless (config('verify_failure_ok')) {
                msg <<EOM;
    Verification failure ok, becuase you've configured fetchware to continue even
    if it cannot verify its downloads. Please reconsider, because mirror and source
    code repos do get hacked. The exception that was caught was:
    [$@]
    EOM
            }
        }
    };
    
    
    hook unarchive => sub {
        # there is nothing to archive due to use of git.
        do_nothing(); # But return the $build_path, which is the cwd().
        my $build_path = $git_repo_dir;
        return $build_path;
    };
    
    # It's a git tag, so it lacks an already generated ./configure, so I must use
    # ./buildconf to generate one. But it won't work on php releases, so I have to
    # force it with --force to convince ./buildconf to run autoconf to generate the
    # ./configure program to configure php for building.
    build_commands './buildconf --force', './configure', 'make';

    # Add any custom configure options that you may want to add to customize
    # your build of php, or control what php extensions get built.
    #configure_options '--whatever you --need ok';
    
    # start() creates a tempdir in most cases this is exactly what you want, but
    # because this Fetchwarefile is using git instead. I don't need to bother with
    # creating a temporary directory.
    hook start => sub {
        # But checkout master anyway that way the repo can be in a known good state
        # so lookup()'s git pull can succeed.
        run_prog('git checkout master');
    };
    
    
    # Switch the local php repo back to the master branch to make using it less
    # crazy. Furthermore, when using git pull to update the repo git uses what
    # branch your on, and if I've checked out a tag, I'm not actually on a branch
    # anymore; therefore, I must switch back to master, so that the git pull when
    # this fetchwarefile is run again will still work.
    hook end => sub {
        run_prog('git checkout master');
    };

MariaDB Database

This example MariaDB Fetchwarefile parses the MariaDB download page to determine what the latest version is based on what filter option you set up. Once this is determined, the download path is created based on the weird path that MariaDB uses on its mirrors.

Like PHP MariaDB uses some annoying software on their Web site to presumably track downloads. This software makes use of AJAX, which is vastly beyone the capabilities of HTML::TreeBuilder to parse, because it needs a working JavaScript environment. Therefore, the example Fetchwarefile below has no way of verifying the MySQL downloads. This could be fixed by using a Perl Web scraping module that can deal with JavaScript.

    use App::Fetchware;
    
    program 'MariaDB';
    
    # MariaDB uses ccache, which wants to create a ~/.ccache cache, which it can't
    # do when it's running as nobody, so use a real user account to ensure ccache
    # has a cache directory it can write to.
    user 'dly';
    
    lookup_url 'https://downloads.mariadb.org/';
    
    # Below are the two USA mirrors where I live. Customize them as you need based
    # on the mirrors listed on the download page (https://downloads.mariadb.org/ and
    # then click on which version you want, and then click on the various mirrors
    # by country. All you need is the scheme (ftp:// or http:// part) and the
    # hostname without a slash (ftp.osuosl.org or mirror.jmu.edu). Not the full path
    # for each mirror.
    mirror 'http://ftp.osuosl.org';
    mirror 'http://mirror.jmu.edu';
    
    # The filter option is key to the custom lookup hook working correctly. It must
    # represent the text that corresponds to the latest GA release of MariaDB
    # available. It should be 'Download 5.5' for 5.5 or 'Download 10.0' for the
    # newver but not GA 10.0 version of MariaDB.
    filter 'Download 5.5';
    
    hook lookup => sub {
        vmsg "Downloading HTML download page listing MariaDB releases.";
        my $dir_list = http_download_dirlist(config('lookup_url'));
    
        vmsg "Parsing HTML page listing MariaDB releases.";
        my $tree = HTML::TreeBuilder->new_from_content($dir_list);
    
        # This parsing code assumes that the latest version of php is the first one
        # we find, which seems like a dependency that's unlikely to change.
        my @version_number;
        $tree->look_down(
            _tag => 'a',
            sub {
                my $h = shift;
                
                my $link = $h->as_text();
    
                # Find the filter which should be "Download\s[LATESTVERSION]"
                my $filter = config('filter');
                if ($link =~ /$filter/) {
                    # Parse out the version number.
                    # It's just the second space separated field.
                    push @version_number, (split ' ', $link)[1];
                }
            }
        );
    
        # Delete the $tree, so perl can garbage collect it.
        $tree = $tree->delete;
    
        # Only one version should be found.
        die <<EOD if @version_number > 1;
    mariaDB.Fetchwarefile: multiple version numbers detected. You should probably
    refine your filter option and try again. Filter [@{[config('filter')]}].
    Versions found [@version_number].
    EOD
    
        # Construct a download path using $version_number[0].
        my $filename = 'mariadb-' . $version_number[0] . '.tar.gz';
    
        # Return a proper $download_path, so That I do not have to hook download(),
        # but can reuse Fetchware's download() subroutine.
        my $weird_prefix = '/mariadb-' . $version_number[0] . '/kvm-tarbake-jaunty-x86/';
        my $download_path = '/pub/mariadb' . $weird_prefix .$filename;
        return $download_path;
    };
    
    # Make verify() failing to verify MariaDB ok, because parsing out the MD5 sum
    # would require a Web scraper that supports javascript, which HTML::TreeBuilder
    # obviously does not.
    verify_failure_ok 'On';
    
    # Use build_commands to configure fetchware to use MariaDB's BUILD script to
    # build it. See https://mariadb.com/kb/en/generic-build-instructions/ for
    # instructions on the different BUILD  cmake scripts that are available.
    build_commands 'BUILD/compile-pentium64-max';
    
    # Use install_commands to tell fetchware how to install it. I could leave this
    # out, but it nicely documents what command is needed to install MariaDB
    # properly.
    install_commands 'make install';

PostgreSQL Database

Below is a example Fetchwarefile that overrides lookup() to determine the latest version, but manages to avoid overriding anything else. It uses the same style as the rest downloading an HTML page that lists the version numbers on it somewhere. Then it parses the HTML with HTML::TreeBuilder. It populates an array, and then uses App::Fetchware's lookup_by_versionstring() to determine which version is the latest one. This is then concatenated with a bunch of other stuff to determine the $download_path.

MD5 verification is supported by simply specifying a md5_url option, because by default fetchware uses the lookup_url to determine where to download the md5sum from, but that won't work with PostgreSQL, because it's download system has the md5sum on the download mirror instead of the lookup_url.

    use App::Fetchware qw(:DEFAULT :OVERRIDE_LOOKUP);
    use App::Fetchware::Util ':UTIL';
    
    use HTML::TreeBuilder;
    
    program 'postgres';
    
    # The Postgres file browser URL lists the available versions of Postgres.
    lookup_url 'http://www.postgresql.org/ftp/source/';
    
    # Mirror URL where the file browser links to download them from.
    my $mirror = 'http://ftp.postgresql.org';
    mirror $mirror;
    
    # The Postgres file browser URL that is used for the lookup_url lists version
    # numbers of Postgres like v9.3.0. this lookup hook parses out the list of
    # theses numbers, determines the latest one, and constructs a $download_path to
    # return for download to use to download based on what I set my mirror to.
    hook lookup => sub {
        my $dir_list = no_mirror_download_dirlist(config('lookup_url'));
    
        my $tree = HTML::TreeBuilder->new_from_content($dir_list);
    
        # Parse out version number directories.
        my @ver_nums;
        my @list_context = $tree->look_down(
            _tag => 'a',
            sub {
                my $h = shift;
    
                my $link = $h->as_text();
    
                # Is this link a version number or something to ignore?
                if ($link =~ /^v\d+\.\d+(.\d+)?$/) {
                    # skip version numbers that are beta's, alpha's or release
                    # candidates (rc).
                    return if $link =~ /beta|alpha|rc/i;
                    # Strip useless "v" that just gets in the way later when I
                    # create the $download_path.
                    $link =~ s/^v//;
                    push @ver_nums, $link;
                }
            }
        );
    
        # Turn @ver_num into the array of arrays that lookup_by_versionstring()
        # needs its arguments to be in.
        my $directory_listing = do {
            my $arrayref_of_arrays_directory_listing = [];
            for my $ver_num (@ver_nums) {
                push @$arrayref_of_arrays_directory_listing,
                    [$ver_num];
            }
            $arrayref_of_arrays_directory_listing;
        };
        # Find latest version.
        my $latest_ver = lookup_by_versionstring($directory_listing);
    
        # Return $download_path.
        my $download_path = '/pub/source/'. "v$latest_ver->[0][0]" .
            "/postgresql-$latest_ver->[0][0].tar.bz2";
        return $download_path;
    };
    
    # MD5sums are stored on the download site, so use them to verify the package.
    verify_method 'md5';
    # But they are *not* stored on the original "lookup_url" site, so I must provide
    # a md5_url pointing to the download site.
    md5_url $mirror;

FAQ ^

Why doesn't fetchware and App::Fetchware use OO or Moose?

One of my goals for fetchware was that its guts be pragmatic. I wanted it to consist of a bunch of subroutines that simply get executed in a specific order. And these subroutines should be small and do one and only one thing to make understanding them easier and to make testing them a breeze.

OO is awesome for massive systems that need to take advandtage of inheritance or perhaps composition. But fetchware is small and simple, and I wanted its implementation to also be small and simple. It is mostly just two four thousand line programs with some smaller utility files. If it were OO there would be half or even a whole dozen number of smaller files, and they would have complicated relationships with each other. I did not want to bother with needless abstration that would just make fetchware more complicated. It is a simple command line program, so it should be written as one.

Moose was out, because I don't need any of its amazing features. I could use those amazing features, but fetchware's simple nature does not demand them. Also, unless Moose and its large chunk of dependencies are in your OS's file system cache, Moose based command line apps take a few seconds to start, because perl has to do a bunch of IO on slow disks to read in Moose and its dependencies. I don't want to waste those few seconds. Plus fetchware is a program not intended for use by experienced Perl developers like dzil is, which does use Moose, and has a few second start up overhead, which is acceptable to its author and maintainer. I also use it, and I'm ok with it, but those few seconds might be off putting to users who have no Perl or Moose knowledge.

What's up with the insane fetchware extension mechanism?

Extension systems are always insane. dzil, for example, uses a configuration file where you list names of dzil plugins, and for each name dzil loads that plugin, and figures out what dzil roles it does, then when dzil executes any commands you give it, dzil executes all of those roles that the plugins registered inside you configuration file.

I wanted a configuration file free of declaring what plugins you're using. I wanted it to be easy to use. dzil is for Perl programmers, so it requiring some knowledge of Perl and Moose is ok. But fetchware is for end users or perhaps system administrators not Perl programmers, so something easier is needed.

The extension mechanism was design for ease of use by people who use your fetchware extension. And it is. Just "use" whatever fetchware extension you want in your Fetchwarefile, and then supply whatever configuration options you need.

This extension mechanism is also very easy for Perl programmers, because you're basically subclassing App::Fetchware, only you do it using App::Fetchware::ExportAPI and App::Fetchware::CreateConfigOptions. See section "Implement your fetchware extension." for full details.

How do I fix the verification failed error.

Fetchware is designed to always attempt to verify the software archives it downloads even if you failed to configure fetchware's verification settings. It will try to guess what those setting should be using simple heuristics. First it will try gpg verificaton, then sha1 verification, and finally md5 verification. If all fail, then fetchware exit failure with an appropriate error message.

When you get this error message read fetchware's documentation on how to set this up.

How do I make fetchware log to a file instead of STDOUT?

You can't fetchware does not have any log file support. However, you can simply redirect STDOUT to a file to make your shell redirect STDOUT to a file for you.

    fetchware install <some-program.Fetchwarefile> > fetchware.log

This would not prevent any error messages from STDERR being printed to your screen for that:

    fetchware install <some-program.Fetchwarefile> 2>&1 fetchware.log

And to throw away all messages use:

    fetchware -q install <some-progra.Fetchwarefile>

or use the shell

    fetchware install <some-program.Fetchwarefile 2>&1 /dev/null

Why don't you use Crypt::OpenPGP instead of the gpg command line program?

I tried to use Crypt::OpenPGP, but I couldn't get it to work. And getting gpg to work was a breeze after digging through its manpage to find the right command line options that did what I need it to.

Also, unfortunately Crypt::OpenPGP is buggy, out-of-date, and seems to have lost another maintainer. If it ever gets another maintainer, who fixes the newer bugs, perhaps I'll add support for Crypt::OpenPGP again. Because of how fetchware works it needs to use supported but not popular options of Crypt::OpenPGP, which may be where the bugs preventing it from working reside.

Supporting Crypt::OpenPGP is still on my TODO list. It's just not very high on that list. Patches are welcome to add support for it, and the old code is still there commented out, but it needs updating if anyone is interested.

In the meantime if you're on Windows without simple access to a gpg command line program, try installing gpg from the gpg4win project, which packages up gpg and a bunch of other tools for easier use on Windows.

Does fetchware support Windows?

Yes and no. I intend to support Windows, but right now I'm running Linux, and my Windows virtual machine is broken, so I can't easily test it on Windows. The codebase makes heavy use of File::Spec and Path::Class, so all of its file operations should work on Windows.

I currently have not tested fetchware on Windows. There are probably some test failures on Windows, but Windows support should be just a few patches away.

So the real answer is no, but I intend to support Windows in a future release.

How do I configure a HTTP proxy for use with Fetchware?

Fetchware uses Perl's HTTP::Tiny module to download files over HTTP. So, all you need to do is configure HTTP::Tiny for use with proxies. This is done with environment variables that can easily be set from your shell.

Set http_proxy in the format http://host:port. You can do this permanantly for this session with:

    export http_proxy='http://example.com:8080'

Or once just for this invocation of fetchware:

    http_proxy='http://example.com:8080 fetchware new

See your OS's shell for more details regarding using and exporting environment variables.

SECURITY ^

App::Fetchware is inherently insecure, because its Fetchwarefile's are executable Perl code that actually is executed by Perl. These Fetchwarefiles are capable of doing everything someone can program Perl to do. This is why App::Fetchware will refuse to execute any Fetchwarefile's that are writable by anyone other than the owner who is executing them. It will also exit with a fatal error if you try to use a Fetchwarefile that is not the same user as the one who is running fetchware. These saftey measures help prevent fetchware being abused to get unauthorized code executed on your computer.

App::Fetchware also features the user configuration option that tells fetchware what user you want fetchware to drop privileges to when it does everything but install (install()) and clean up (end()). The configuration option does not tell fetchware to turn on the drop privelege code; that code is always on, but just uses the fairly ubuiquitous nobody user by default. This feature requires the OS to be some version of Unix, because Windows and other OSes do not support the same fork()ing method of limiting what processes can do. On non-Unix OSes, fetchware won't fork() or try to use some other way of dropping privileges. It only does it on Unix. If you use some version of Unix, and do not want fetchware to drop privileges, then specify the stay_root configuration option.

ERRORS ^

App::Fetchware does not return any error codes; instead, all errors are die()'d if it's App::Fetchware::Config's error, or croak()'d if its the caller's fault. These exceptions are short paragraphs that give full details about the error instead of the vague one liner that perl's own errors give.

BUGS ^

The official bug tracker for fetchware is its github issues page.

RESTRICTIONS ^

AUTHOR ^

David Yingling <deeelwy@gmail.com>

COPYRIGHT AND LICENSE ^

This software is copyright (c) 2013 by David Yingling.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

syntax highlighting: