The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
upfiles-11
River stage zero No dependents
++
/ upfiles

NAME

upfiles -- upload files to an FTP server, for push mirroring

SYNOPSIS

 upfiles [--options] [filename...]

DESCRIPTION

Upfiles uploads changed files from your local disk to an FTP server, for a simple kind of "push" mirroring.

Create files locally with the same directory structure as the target, and in a ~/.upfiles.conf file give the locations,

    upfiles (local  => '/my/directory',
             remote => 'ftp://fred@some-server.org/pub/fred');

This is actually Perl code, so you can put comment lines with #, write some conditionals, use $ENV{HOME}, etc. Then to upload run

    upfiles

Or to upload just some selected files (the local filenames),

    upfiles /my/directory/foo.txt /my/directory/src/xyzzy.pl

Your username on the remote system is in the ftp:// remote URL. A password is taken from ~/.netrc the same as for the ftp program and other programs. See netrc(5) or Net::Netrc for the format of that file.

upfiles records what has been sent in an SQLite database file .upfiles.sqdb in each local toplevel directory, for example /my/directory/.upfiles.sqdb. Local changes are identified by comparing file mtimes and sizes against the database. This is faster than asking the remote server what it's got.

For convenience some local files are always excluded from the upload. Currently these are

    .upfiles.sqdb    from upfiles itself
    foo~             Emacs backups
    #foo#            Emacs autosaves
    .#foo            Emacs lockfiles

Files are uploaded one by one. The upload goes first to a temporary file and is then renamed. This means an incomplete file isn't left if the connection is lost or upfiles is killed during transfer. Temporary files are noted in the database and leftovers are deleted on the next upfiles run if necessary.

File modification times are copied to the server if it has the draft standard MFMT or the common SITE UTIME extension (2-arg or 5-arg).

Plain RFC959 ftp doesn't have a notion of symlinks or hard links so upfiles follows any local links to actual content to upload. (Perhaps in the future the SITE SYMLINK extension could be used if available.)

CONFIGURATION

Each upfiles call in ~/.upfiles.conf takes the following parameters,

local (string)

The local directory to upload from.

remote (string)

The remote FTP server to upload to, as a URL. The path in the URL is the target directory, and if your username on the remote machine is not the same as your local username then include it with "@" syntax, like

    remote => 'ftp://fred@some-server.org/pub/fred',
exclude_regexps (arrayref of regexps)

Additional filenames to exclude. For example to exclude a local Makefile

    upfiles (local => '/my/directory',
             remote => 'ftp://some-server.org/pub/fred',
             exclude_regexps => [ qr{/(^|/)[Mm]akefile$} ]);
copy_utime (0, 1, default "if_possible")

Whether to copy file modification times to the server with the MFMT or SITE UTIME command. The default "if_possible" means do so if the server supports it. 0 means don't try. 1 means it must work.

COMMAND-LINE OPTIONS

The command line options are

-n, --dry-run

Show what would be uploaded or deleted on the server, but don't actually do anything.

    upfiles -n
--help

Print some brief help information.

-V, --verbose, --verbose=N

Print some diagnostics about what's being done. With --verbose=2 or --verbose=3 print some technical details too.

    upfiles --verbose
--version

Print the upfiles program version number. With --verbose=2 also print the version numbers of some modules used.

FILES

~/.upfiles.conf

Configuration file.

~/.netrc

FTP password file.

/some/local/dir/.upfiles.sqdb

SQLite database of information about what has been sent.

Upfiles determines the home directory ~ using File::HomeDir. Net::Netrc has something similar for the .netrc file, and looks also for _netrc.

BUGS

Changing a local file from a file to a directory or vice versa probably doesn't work very well. Remove it and upload, then create the new and upload that.

FTP requires a couple of round trip command/responses to the server for every file. When uploading many small files something streaming or parallel would be faster. The temp file and rename adds a round trip too, but is desirable so anyone looking doesn't see a half file. Perhaps an option could turn this off if not needed (such as upfiles to a remote backup).

The temporary files are named using the local $$ PID added to the target filename. This is enough to protect against simultaneous uploads from the same source machine, but potentially unsafe if you're networked and are foolish enough to upfiles the same tree simultaneously from two different source machines. STOU would guarantee uniqueness, but does it leave a window while the name comes back which if interrupted could leave the file created but unknown? Net::FTP put_unique() doesn't return the name until after transfer too.

Filenames containing \r or \n cannot be sent by FTP protocol and Upfiles will refuse to operate on them.

There's a small chance of a 2-arg SITE UTIME attempt looking like a 5-arg to a pre-1.0.33 pure-ftpd server (of circa 2011), if the filename happens to be dates and "UTC". Perhaps more care could be taken to identify the SITE UTIME style the server expects. In practice such filenames should be unlikely, and there's no problem with recent pure-ftpd which has MDTM.

SEE ALSO

Net::FTP, netrc(5), Net::Netrc, DBD::SQLite

sitecopy(1), ftpmirror(1), ftp-upload(1), rsync(1)

HOME PAGE

http://user42.tuxfamily.org/upfiles/index.html

(Upfiles is good for uploading to tuxfamily.)

LICENSE

Copyright 2009, 2010, 2011, 2012, 2013, 2014, 2015 Kevin Ryde

Upfiles is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.

Upfiles is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Upfiles. If not, see http://www.gnu.org/licenses/.