#!perl -w
# Copyright 2009, 2010, 2011, 2012, 2013, 2014, 2015 Kevin Ryde
# This file is part of Upfiles.
#
# Upfiles is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 3, or (at your option) any later
# version.
#
# Upfiles is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along
# with Upfiles. If not, see <http://www.gnu.org/licenses/>.
use 5.010;
use strict;
use warnings;
use App::Upfiles;
our $VERSION = 11;
my $upf = App::Upfiles->new;
exit $upf->command_line;
__END__
=for stopwords upfiles Upfiles username SQLite toplevel mtimes symlinks arrayref filenames versa PID filename tuxfamily Ryde
=head1 NAME
upfiles -- upload files to an FTP server, for push mirroring
=head1 SYNOPSIS
upfiles [--options] [filename...]
=head1 DESCRIPTION
Upfiles uploads changed files from your local disk to an FTP server, for a
simple kind of "push" mirroring.
Create files locally with the same directory structure as the target, and in
a F<~/.upfiles.conf> file give the locations,
upfiles (local => '/my/directory',
remote => 'ftp://fred@some-server.org/pub/fred');
This is actually Perl code, so you can put comment lines with C<#>, write
some conditionals, use C<< $ENV{HOME} >>, etc. Then to upload run
upfiles
Or to upload just some selected files (the local filenames),
upfiles /my/directory/foo.txt /my/directory/src/xyzzy.pl
Your username on the remote system is in the C<ftp://> remote URL.
A password is taken from C<~/.netrc> the same as for the C<ftp> program and
other programs. See L<netrc(5)> or L<Net::Netrc> for the format of that
file.
B<upfiles> records what has been sent in an SQLite database file
F<.upfiles.sqdb> in each local toplevel directory, for example
F</my/directory/.upfiles.sqdb>. Local changes are identified by comparing
file mtimes and sizes against the database. This is faster than asking the
remote server what it's got.
For convenience some local files are always excluded from the upload.
Currently these are
.upfiles.sqdb from upfiles itself
foo~ Emacs backups
#foo# Emacs autosaves
.#foo Emacs lockfiles
Files are uploaded one by one. The upload goes first to a temporary file
and is then renamed. This means an incomplete file isn't left if the
connection is lost or upfiles is killed during transfer. Temporary files
are noted in the database and leftovers are deleted on the next upfiles run
if necessary.
File modification times are copied to the server if it has the draft
standard C<MFMT> or the common C<SITE UTIME> extension (2-arg or 5-arg).
Plain RFC959 ftp doesn't have a notion of symlinks or hard links so
C<upfiles> follows any local links to actual content to upload. (Perhaps in
the future the C<SITE SYMLINK> extension could be used if available.)
=head1 CONFIGURATION
Each C<upfiles> call in F<~/.upfiles.conf> takes the following parameters,
=over 4
=item C<local> (string)
The local directory to upload from.
=item C<remote> (string)
The remote FTP server to upload to, as a URL. The path in the URL is the
target directory, and if your username on the remote machine is not the same
as your local username then include it with "@" syntax, like
remote => 'ftp://fred@some-server.org/pub/fred',
=item C<exclude_regexps> (arrayref of regexps)
Additional filenames to exclude. For example to exclude a local F<Makefile>
upfiles (local => '/my/directory',
remote => 'ftp://some-server.org/pub/fred',
exclude_regexps => [ qr{/(^|/)[Mm]akefile$} ]);
=item C<copy_utime> (0, 1, default C<"if_possible">)
Whether to copy file modification times to the server with the C<MFMT> or
C<SITE UTIME> command. The default C<"if_possible"> means do so if the
server supports it. 0 means don't try. 1 means it must work.
=back
=head1 COMMAND-LINE OPTIONS
The command line options are
=over 4
=item -n, --dry-run
Show what would be uploaded or deleted on the server, but don't actually do
anything.
upfiles -n
=item --help
Print some brief help information.
=item -V, --verbose, --verbose=N
Print some diagnostics about what's being done. With --verbose=2 or
--verbose=3 print some technical details too.
upfiles --verbose
=item --version
Print the upfiles program version number. With C<--verbose=2> also print
the version numbers of some modules used.
=back
=head1 FILES
=over 4
=item F<~/.upfiles.conf>
Configuration file.
=item F<~/.netrc>
FTP password file.
=item F</some/local/dir/.upfiles.sqdb>
SQLite database of information about what has been sent.
=back
Upfiles determines the home directory F<~> using L<File::HomeDir>.
C<Net::Netrc> has something similar for the F<.netrc> file, and looks also
for F<_netrc>.
=head1 BUGS
Changing a local file from a file to a directory or vice versa probably
doesn't work very well. Remove it and upload, then create the new and
upload that.
FTP requires a couple of round trip command/responses to the server for
every file. When uploading many small files something streaming or parallel
would be faster. The temp file and rename adds a round trip too, but is
desirable so anyone looking doesn't see a half file. Perhaps an option
could turn this off if not needed (such as upfiles to a remote backup).
The temporary files are named using the local C<$$> PID added to the target
filename. This is enough to protect against simultaneous uploads from the
same source machine, but potentially unsafe if you're networked and are
foolish enough to C<upfiles> the same tree simultaneously from two different
source machines. C<STOU> would guarantee uniqueness, but does it leave a
window while the name comes back which if interrupted could leave the file
created but unknown? C<Net::FTP> C<put_unique()> doesn't return the name
until after transfer too.
Filenames containing \r or \n cannot be sent by FTP protocol and Upfiles
will refuse to operate on them.
There's a small chance of a 2-arg C<SITE UTIME> attempt looking like a 5-arg
to a pre-1.0.33 pure-ftpd server (of circa 2011), if the filename happens to
be dates and "UTC". Perhaps more care could be taken to identify the C<SITE
UTIME> style the server expects. In practice such filenames should be
unlikely, and there's no problem with recent pure-ftpd which has C<MDTM>.
=head1 SEE ALSO
L<Net::FTP>, L<netrc(5)>, L<Net::Netrc>, L<DBD::SQLite>
L<sitecopy(1)>, L<ftpmirror(1)>, L<ftp-upload(1)>, L<rsync(1)>
=head1 HOME PAGE
L<http://user42.tuxfamily.org/upfiles/index.html>
(Upfiles is good for uploading to tuxfamily.)
=head1 LICENSE
Copyright 2009, 2010, 2011, 2012, 2013, 2014, 2015 Kevin Ryde
Upfiles is free software; you can redistribute it and/or modify it under the
terms of the GNU General Public License as published by the Free Software
Foundation; either version 3, or (at your option) any later version.
Upfiles is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
details.
You should have received a copy of the GNU General Public License along with
Upfiles. If not, see L<http://www.gnu.org/licenses/>.
=cut