Home · Authors · Recent · News · Mirrors · FAQ · Feedback

in  
Casiano Rodriguez-Leon > Remote-Use > Remote::Use::Tutorial

Download:
Remote-Use-0.04.tar.gz

Annotate this POD

View/Report Bugs
Source  

NAME ^

Remote::Use::Tutorial - An introduction to Remote::Use

SYNOPSIS ^

  $ cat -n prime3.pl
     1  #!/usr/bin/perl -I../lib -w
     2  # The module Math::Prime::XS isn't installed in the machine
     3  # but will be downloaded from some remote server
     4  use Math::Prime::XS qw{:all};
     5
     6  @all_primes   = primes(9);
     7  print "@all_primes\n";
     8
     9  @range_primes = primes(4, 9);
    10  print "@range_primes\n";

  $ cat -n rsyncconfig
     1  package rsyncconfig; # Configuration file
     2
     3  sub getarg {
     4    my ($class, $self) = @_;
     5
     6    return (
     7      host => 'orion:',
     8      prefix => '/tmp/perl5lib/',
     9      command => 'rsync -i -vaue ssh',
    10      ppmdf => '/tmp/perl5lib/.orion.installed.modules',
    11    );
    12  }
    13
    14  1;


  $ time perl -MRemote::Use=config,rsyncconfig prime3.pl
  receiving file list ... done
  >f+++++++++ XS.so

  sent 42 bytes  received 16141 bytes  10788.67 bytes/sec
  total size is 16043  speedup is 0.99
  receiving file list ... done
  >f+++++++++ XS.bs

  sent 42 bytes  received 94 bytes  90.67 bytes/sec
  total size is 0  speedup is 0.00
  receiving file list ... done
  >f+++++++++ XS.pm

  sent 42 bytes  received 5733 bytes  11550.00 bytes/sec
  total size is 5635  speedup is 0.98
  2 3 5 7
  5 7

  real    0m2.349s
  user    0m0.116s
  sys     0m0.060s

  # Second time: the modules have already been loaded
  $ time perl -MRemote::Use=config,rsyncconfig prime3.pl
  2 3 5 7
  5 7

  real    0m0.066s
  user    0m0.056s
  sys     0m0.008s

SUMMARY AND MOTIVATIONS ^

When I wrote this module I didn't know of the existence of PAR::Repository::Client and PAR::Repository distributions. These distributions - now embedded inside the PAR distribution - broach similar problems.

Look at them first before considering using this one. They provide a solution more robust and cover a range of problems much wider than this distribution.

Remote::Use focuses in the direct loading of modules already installed in some remote server while PAR::Repository::Client and PAR::Repository deal with the use of remote PAR repositories.

There are a few scenarios where this module can still be useful:

Remote::Use provides a way to run a Perl program when some libraries aren't availables at start time. The libraries will be downloaded from a specified server using a specified application that runs on top of some protocol. The clients must be binary compatibles with the server if binary libraries - as is the case of the module Math::Prime::XS used in the SYNOPSIS section example script - are involved. Typical downloaders are scp, rsync or wget but any other suitable alternative like lwp-mirror or Curl can be used. This means that many different protocols can be used for the transference: SSH, SFTP, HTTP, HTTPS, FTP, etc.

This way, the clients download the modules their programs use to some directory. Once the modules are downloaded they will not be downloaded again, unless the modules are removed.

There are many ways of setting a Machine to serve its installed Perl Modules to other clients. The process implies the creation of a file that describes the modules you want to make public. We will consider here only two ways: Setting a server that is accesible via SSH and setting a server that is accesible via HTTP.

The next section deals with the case where the clients have automatic SSH access to the server. See the "APPENDIX: AUTOMATIC AUTHENTICATION" if you aren't familiar with automatic SSH authentication.

Later we will study the case when the server is accesible via HTTP or HTTPS.

After reading these sections the use of Remote::Use with other protocols (FTP, SFTP, etc.) must be straightforward. See also the examples directory in the accompanying distribution. The discussion of a more detailed control of the downloaded files - with special emphasis in how to download the executables associated with some module - is studied in section "DOWNLOADING EXECUTABLES WITH wget".

USING REMOTE MODULES WITH rsync VIA SSH ^

CREATING A PERL PUBLIC MODULES DESCRIPTOR FILE

To use remote modules, the clients need to have accesible a Perl Public Modules Descriptor File (PPMDF). Such file describes (in Perl) the set of Modules in the server that are visible/public to the clients. The script pminstalled.pl that accompanies the distribution of Remote::Use simplifies the process of writing such PPMDF files.

Along this document we will assume that machine nereida is the client and machine orion is the Perl Modules Server. In this particular section we will also assume that automatic SSH autentification has been set up between client and server.

We start by copying the pminstalled.pl script to the server:

  pp2@nereida:$ scp pminstalled.pl orion:
  pminstalled.pl                                                       100% 1551     1.5KB/s   00:00

Of course the transference isn't needed if you already have installed Remote::Use in the Perl Public Modules Server. Now run pminstalled.pl in the server, saving the output in the client:

  pp2@nereida:$ ssh orion perl pminstalled.pl > /tmp/perl5lib/.orion.installed.modules
  Duplicated module 'Test/Builder.pm':
  Is in:
         /usr/local/share/perl/5.8.8/Test/Builder.pm
  and in:
         /usr/local/share/perl/5.8.4/Test/Builder.pm
  only the first will be considered.

  Duplicated module 'Test/Simple.pm':
  Is in:
         /usr/local/share/perl/5.8.8/Test/Simple.pm
  and in:
         /usr/local/share/perl/5.8.4/Test/Simple.pm
  only the first will be considered.

  ..... etc, etc.

  pp2@nereida:$

The execution of pminstalled.pl warns the user about those modules that appear more than once in the server @INC path. The warning message tell us that only the first copy will be transferred to the client.

To redirect the warning messages to a file use the log option of pminstalled.pl:

  pp2@nereida:$ ssh orion perl pminstalled.pl -log /tmp/dups > /tmp/perl5lib/.orion.installed.modules

Now the warning messages are in the file /tmp/dups in the machine orion:

  pp2@nereida:$ ssh orion head /tmp/dups
  Duplicated module 'Test/Builder.pm':
  Is in:
         /usr/local/share/perl/5.8.8/Test/Builder.pm
  and in:
         /usr/local/share/perl/5.8.4/Test/Builder.pm
  only the first will be considered.

  Duplicated module 'Test/Simple.pm':
  Is in:
         /usr/local/share/perl/5.8.8/Test/Simple.pm

Running pminstalled.pl without options send to STDOUT a Perl Public Module Descriptor File (PPMDF). Since STDOUT has been redirected in the former example, we have the PPMDF file /tmp/perl5lib/.orion.installed.modules that describes the set of modules found in the server @INC path. The descriptor file /tmp/perl5lib/.orion.installed.modules is written in Perl itself. It is an array, that for each module found, has a hash entry. See the first lines of such file:

  $ head -19 /tmp/perl5lib/.orion.installed.modules
  (
  'CPAN/Config.pm' => { dir => '/etc/perl', files => [
          '/etc/perl/CPAN/Config.pm' ] },
  'Template.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
          '/usr/local/lib/perl/5.8.8/auto/Template/Stash/XS/XS.so',
          '/usr/local/lib/perl/5.8.8/auto/Template/Stash/XS/XS.bs',
          '/usr/local/lib/perl/5.8.8/Template.pm' ] },
  'IO/Tty.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
          '/usr/local/lib/perl/5.8.8/auto/IO/Tty/Tty.so',
          '/usr/local/lib/perl/5.8.8/auto/IO/Tty/Tty.bs',
          '/usr/local/lib/perl/5.8.8/IO/Tty.pm' ] },
  'IO/Pty.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
          '/usr/local/lib/perl/5.8.8/IO/Pty.pm' ] },
  'IO/Tty/Constant.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
          '/usr/local/lib/perl/5.8.8/IO/Tty/Constant.pm' ] },
  'Math/Prime/XS.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.so',
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.bs',
          '/usr/local/lib/perl/5.8.8/Math/Prime/XS.pm' ] },

Thus, for example for module Math::Prime::XS we have the entry:

  'Math/Prime/XS.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.so',
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.bs',
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/.packlist',
          '/usr/local/lib/perl/5.8.8/Math/Prime/XS.pm' ] },

The key for such entry is the file name 'Math/Prime/XS.pm' associated with the use Math::Prime::XS of the module Math::Prime::XS. The value is a HASH reference that describes what files must be downloaded.

The value associated with the key dir of such hash,

            '/usr/local/lib/perl/5.8.8'

is the path where the file 'Math/Prime/XS.pm' was found. The items in the list corresponding to the key file contain files that are associated with the module and must be downloaded when this modules are used:

          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.so',
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.bs',
          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/.packlist',
          '/usr/local/lib/perl/5.8.8/Math/Prime/XS.pm'

the last element in such list is the full path to the module itself:

          '/usr/local/lib/perl/5.8.8/Math/Prime/XS.pm'

Of course, you can edit and modify such file at your convenience or use your own PPMDF generator instead of pminstalled.pl.

If you want to make public modules that aren't in the official @INC path, just add the corresponding -I options to the perl interpreter executing pminstalled.pl:

 $ ssh orion perl -I/home/casiano/public_html/cpan pminstalled.pl \
             > /tmp/perl5lib/.orion.plus.public.installed.modules
  Duplicated module 'Parse/Eyapp.pm':
  Is in:
         /home/casiano/public_html/cpan/Parse/Eyapp.pm
  and in:
         /usr/local/share/perl/5.8.8/Parse/Eyapp.pm
  only the first will be considered.

  ......, etc, etc.

  $

Now the PPMDF generated also contains the modules in /home/casiano/public_html/cpan:

  pp2@nereida:$ head /tmp/perl5lib/.orion.plus.public.installed.modules
  (
  'Trivial.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Trivial.pm' ] },
  'Tintin/Trivial.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Tintin/Trivial.pm' ] },
  'Parse/Eyapp.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Parse/Eyapp.pm' ] },
  'Parse/Eyapp/Lalr.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Parse/Eyapp/Lalr.pm' ] },
  'Parse/Eyapp/YATW.pm' => { dir => '/home/casiano/public_html/cpan', files => [

What if we want an entirely different search path alternative to @INC? For that we execute pminstalled.pl with the list of directories where to look at:

 $ ssh orion perl pminstalled.pl /home/casiano/public_html/cpan -o /home/casiano/public_html/.orion.via.web

The option -o of pminstalled.pl saves the output in the specified file /tmp/.orion.via.web. Since we are using a ssh command the file was saved in the remote machine:

  $ ssh orion head /home/casiano/public_html/.orion.via.web
  (
  'Trivial.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Trivial.pm' ] },
  'Tintin/Trivial.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Tintin/Trivial.pm' ] },
  'Parse/Eyapp.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Parse/Eyapp.pm' ] },
  'Parse/Eyapp/Lalr.pm' => { dir => '/home/casiano/public_html/cpan', files => [
          '/home/casiano/public_html/cpan/Parse/Eyapp/Lalr.pm' ] },
  'Parse/Eyapp/YATW.pm' => { dir => '/home/casiano/public_html/cpan', files => [
  pp2@nereida:~$

The PPDF file /home/casiano/public_html/.orion.via.web describes the modules in the directory

                       /home/casiano/public_html/cpan

See the tree hierarchy of orion:/home/casiano/public_html/cpan:

  $ ssh orion tree /home/casiano/public_html/cpan
  /home/casiano/public_html/cpan
  |-- Math
  |   `-- Prime
  |       `-- XS.pm
  |-- Parse
  |   |-- Eyapp
  |   |   |-- Base.pm
  |   |   |-- Driver.pm
  |   |   |-- Grammar.pm
  |   |   |-- Lalr.pm
  |   |   |-- Node.pm
  |   |   |-- Options.pm
  |   |   |-- Output.pm
  |   |   |-- Parse.pm
  |   |   |-- Scope.pm
  |   |   |-- Treeregexp.pm
  |   |   |-- YATW.pm
  |   |   `-- _TreeregexpSupport.pm
  |   `-- Eyapp.pm
  |-- Tintin
  |   `-- Trivial.pm
  |-- Trivial.pm
  |-- auto
  |   `-- Math
  |       `-- Prime
  |           `-- XS
  |               |-- XS.bs
  |               `-- XS.so
  `-- bin
      |-- eyapp
      `-- treereg

  10 directories, 20 files

LOADING REMOTE MODULES WITH rsync

Once we have saved the PPMDF somewhere inside the client machine, we have to create a configuration package. Such configuration package is a Perl package describing the connection with the Perl Modules Server. While the PPMDF file tell us where are the files to transfer, the configuration package says how they will be transferred. The configuration package specifies, among other things, where the PPMDF file is and what application will be used for the transference of files.

In this section we will use rsync, an application that synchronizes files and directories between two machines optimizing the volume of data to transfer using delta encoding.

Start by writing a configuration file similar to this:

  pp2@nereida:~/LRemoteUse/examples$ cat -n rsyncconfig
     1  package rsyncconfig;
     2
     3  sub getarg {
     4    my ($class, $self) = @_;
     5
     6    return (
     7      host => 'orion:',
     8      prefix => '/tmp/perl5lib/',
     9      command => 'rsync -i -vaue ssh',
    10      ppmdf => '/tmp/perl5lib/.orion.installed.modules',
    11    );
    12  }
    13
    14  1;

The configuration file must have a subroutine named getarg. Such subroutine sets the attributes of the Remote::Use object that lead the behavior of Remote::Use during downloading. It receives as arguments the configuration package identifier and a reference to the Remote::Use object. Let us describe each of the attributes returned by getarg:

RUNNING A SCRIPT THAT MAKES USE OF REMOTE MODULES

Once we have the PPMDF file /tmp/perl5lib/.orion.installed.modules and the configuration package rsyncconfig written we can make automatic the downloading of remote modules:

  pp2@nereida:~/LRemoteUse/examples$ cat -n prime2.pl
     1  #!/usr/bin/perl -I../lib -w
     2  use Remote::Use config => 'rsyncconfig';
     3  use Math::Prime::XS qw{:all};
     4
     5  @all_primes   = primes(9);
     6  print "@all_primes\n";
     7
     8  @range_primes = primes(4, 9);
     9  print "@range_primes\n";

Notice line 2 in the listing above: Remote::Use push an object reference into the @INC array. After all the previous paths in @INC have been searched, the Remote::Use::INC method is executed. Such method attempts to open a connection to the remote server and to download the required files. Therefore, for performance reasons, it is convenient to be sure that the Remote::Use object reference is the last in the @INC list.

The first time we run prime2.pl the files specified in the PPMDF for Math::Prime::XS are downloaded:

  pp2@nereida:~/LRemoteUse/examples$ time prime2.pl
  receiving file list ... done
  >f+++++++++ XS.so

  sent 42 bytes  received 16141 bytes  10788.67 bytes/sec
  total size is 16043  speedup is 0.99
  receiving file list ... done
  >f+++++++++ XS.bs

  sent 42 bytes  received 94 bytes  272.00 bytes/sec
  total size is 0  speedup is 0.00
  receiving file list ... done
  >f+++++++++ XS.pm

  sent 42 bytes  received 5733 bytes  3850.00 bytes/sec
  total size is 5635  speedup is 0.98
  2 3 5 7
  5 7

  real    0m2.506s
  user    0m0.148s
  sys     0m0.052s

Since the files are now cached, the second run does not involves additional overhead:

  pp2@nereida:~/LRemoteUse/examples$ time prime2.pl
  2 3 5 7
  5 7

  real    0m0.073s
  user    0m0.052s
  sys     0m0.012s

An alternative to introduce the explicit use of Remote::Use inside the script is to call the perl interpreter from the command line with the -M option:

  pp2@nereida:~/LRemoteUse/examples$ perl -MRemote::Use=config,rsyncconfig prime3.pl

Another alternative to the configuration file is to explicitly set the options in the use directive like in:

  pp2@nereida:~/LRemoteUse/examples$ cat -n primeex.pl
     1  #!/usr/bin/perl -I../lib -w
     2  use Remote::Use
     3      host => 'orion:',
     4      prefix => '/tmp/perl5lib/',
     5      command => 'rsync -i -vaue ssh',
     6      ppmdf => '/tmp/perl5lib/.orion.installed.modules',
     7  ;
     8  use Math::Prime::XS qw{:all};
     9
    10  @all_primes   = primes(9);
    11  print "@all_primes\n";
    12
    13  @range_primes = primes(4, 9);
    14  print "@range_primes\n";

For an example that uses scp instead of rsync see the files examples/reqprimewithscp.pl and examples/rsyncconfigwithscp accompanying the distribution of Remote::Use.

USING REMOTE MODULES WITH wget VIA HTTP/HTTPS/FTP ^

The program wget is a non-interactive free software application for retrieving files using HTTP, HTTPS and FTP protocols. It runs on most UNIX-like operating systems as well as Microsoft Windows, Mac OS X, OpenVMS and AmigaOS. An alternative to wget is cURL.

To manually download a file like XS.pm using wget we use a command like:

  $ wget -o /tmp/wget.log http://orion.pcg.ull.es/~casiano/cpan/Math/Prime/XS.pm -O XS.pm

This will download the file /home/casiano/public_html/cpan/Math/Prime/XS.pm from the machine orion.pcg.ull.es using the HTTP protocol. The downloaded file will be stored as XS.pm in the current directory. The application wget not only supports HTTP transfers but also HTTPS and FTP. cURL is even richer, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE.

The way of working is similar to rsync but a few things change. We will see them in detail in the following paragraphs.

THE -relative OPTION OF pminstalled.pl

The pminstalled.pl script has the option -relative somepath. If used it will produce a PPMDF in which the prefix somepath is supressed from the dir attribute. The following comand:

 $ ssh orion perl pminstalled.pl -r /home/casiano/public_html/cpan /home/casiano/public_html/cpan > /tmp/perl5lib/.orion.via.web

executes pminstalled.pl in the server orion via ssh. The second /home/casiano/public_html/cpan argument tells pminstalled.pl to publish in the PPMDF file the modules in that directory /home/casiano/public_html/cpan (but not those in @INC). The consequence of using -r /home/casiano/public_html/cpan is that the dir entries in the PPMDF file /tmp/perl5lib/.orion.via.web are empty:

 $ cat -n /tmp/perl5lib/.orion.via.web
     1  (
     2  'Trivial.pm' => { dir => '', files => [
     3          '/Trivial.pm' ] },
     4  'Tintin/Trivial.pm' => { dir => '', files => [
     5          '/Tintin/Trivial.pm' ] },
     6  'Parse/Eyapp.pm' => { dir => '', files => [
     7          '/Parse/Eyapp.pm' ] },
     8  'Parse/Eyapp/Lalr.pm' => { dir => '', files => [
     9          '/Parse/Eyapp/Lalr.pm' ] },
    10  'Parse/Eyapp/YATW.pm' => { dir => '', files => [
    11          '/Parse/Eyapp/YATW.pm' ] },
    12  'Parse/Eyapp/Treeregexp.pm' => { dir => '', files => [
    13          '/Parse/Eyapp/Treeregexp.pm' ] },
    14  'Parse/Eyapp/Parse.pm' => { dir => '', files => [
    15          '/Parse/Eyapp/Parse.pm' ] },
    16  'Parse/Eyapp/Scope.pm' => { dir => '', files => [
    17          '/Parse/Eyapp/Scope.pm' ] },
    18  'Parse/Eyapp/Options.pm' => { dir => '', files => [
    19          '/Parse/Eyapp/Options.pm' ] },
    20  'Parse/Eyapp/Output.pm' => { dir => '', files => [
    21          '/Parse/Eyapp/Output.pm' ] },
    22  'Parse/Eyapp/Node.pm' => { dir => '', files => [
    23          '/Parse/Eyapp/Node.pm' ] },
    24  'Parse/Eyapp/Grammar.pm' => { dir => '', files => [
    25          '/Parse/Eyapp/Grammar.pm' ] },
    26  'Parse/Eyapp/Driver.pm' => { dir => '', files => [
    27          '/Parse/Eyapp/Driver.pm' ] },
    28  'Parse/Eyapp/Base.pm' => { dir => '', files => [
    29          '/Parse/Eyapp/Base.pm' ] },
    30  'Parse/Eyapp/_TreeregexpSupport.pm' => { dir => '', files => [
    31          '/Parse/Eyapp/_TreeregexpSupport.pm' ] },
    32  'Math/Prime/XS.pm' => { dir => '', files => [
    33          '/auto/Math/Prime/XS/XS.bs',
    34          '/auto/Math/Prime/XS/XS.so',
    35          '/Math/Prime/XS.pm' ] },
    36  );

That mostly maches the contents of the directory /home/casiano/public_html/cpan/ in the server machine (orion):

  casiano@orion:~$ tree /home/casiano/public_html/cpan/
  /home/casiano/public_html/cpan/
  |-- Math
  |   `-- Prime
  |       `-- XS.pm
  |-- Parse
  |   |-- Eyapp
  |   |   |-- Base.pm
  |   |   |-- Driver.pm
  |   |   |-- Grammar.pm
  |   |   |-- Lalr.pm
  |   |   |-- Node.pm
  |   |   |-- Options.pm
  |   |   |-- Output.pm
  |   |   |-- Parse.pm
  |   |   |-- Scope.pm
  |   |   |-- Treeregexp.pm
  |   |   |-- YATW.pm
  |   |   `-- _TreeregexpSupport.pm
  |   `-- Eyapp.pm
  |-- Tintin
  |   `-- Trivial.pm
  |-- Trivial.pm
  |-- auto
  |   `-- Math
  |       `-- Prime
  |           `-- XS
  |               |-- XS.bs
  |               `-- XS.so
  `-- bin
      |-- eyapp
      `-- treereg

A CONFIGURATION FILE FOR wget

Let us now write a program that makes use of this PPMDF file to download the modules it needs:

  pp2@nereida:~/LRemoteUse/examples$ cat -n prime2wget.pl
     1  #!/usr/bin/perl -I../lib -w
     2  use Remote::Use config => 'wgetconfig';
     3  use Math::Prime::XS qw{:all};
     4
     5  @all_primes   = primes(9);
     6  print "@all_primes\n";
     7
     8  @range_primes = primes(4, 9);
     9  print "@range_primes\n";

The module Math::Prime::XS (line 3) is in the remote machine (orion). The configuration file now makes use of the program wget to download the files;

  pp2@nereida:~/LRemoteUse/examples$ cat -n wgetconfig
     1  package wgetconfig;
     2  use strict;
     3  use warnings;
     4
     5  sub getarg {
     6    return (
     7      command => 'wget -o /tmp/wget.log',
     8      commandoptions => '-O',
     9      host => 'http://orion.pcg.ull.es/~casiano/cpan',
    10      prefix => '/tmp/perl5lib/',
    11      ppmdf => '/tmp/perl5lib/.orion.via.web',
    12    );
    13  }
    14
    15  1;

Let us see the meaning of the different arguments of getarg:

DOWNLOADING AND RUNNING WITH wget

When prime2wget.pl it is executed it silently downloads the files and produces the correct output:

  pp2@nereida:~/LRemoteUse/examples$ time prime2wget.pl
  2 3 5 7
  5 7

  real    0m0.084s
  user    0m0.052s
  sys     0m0.028s

The second time the module has been loaded and it takes less time:

  pp2@nereida:~/LRemoteUse/examples$ time prime2wget.pl
  2 3 5 7
  5 7

  real    0m0.044s
  user    0m0.024s
  sys     0m0.020s

The option -o /tmp/wget.log redirects the log messages to the file /tmp/wget.log:

  pp2@nereida:~/LRemoteUse/examples$ cat /tmp/wget.log
  --12:04:24--  http://orion.pcg.ull.es/~casiano/cpan/Math/Prime/XS.pm
             => `/tmp/perl5lib//Math/Prime/XS.pm'
  Resolviendo orion.pcg.ull.es... 193.145.105.17
  Connecting to orion.pcg.ull.es|193.145.105.17|:80... conectado.
  Petición HTTP enviada, esperando respuesta... 200 OK
  Longitud: 5,635 (5.5K) [text/x-perl]

      0K .....                                      100%    7.29 MB/s

  12:04:24 (7.29 MB/s) - `/tmp/perl5lib//Math/Prime/XS.pm' saved [5635/5635]

The files are now in the directory /tmp/perl5lib/:

  pp2@nereida:~/LRemoteUse/examples$ tree /tmp/perl5lib/
  /tmp/perl5lib/
  `-- files
      |-- Math
      |   `-- Prime
      |       `-- XS.pm
      `-- auto
          `-- Math
              `-- Prime
                  `-- XS
                      |-- XS.bs
                      `-- XS.so

  7 directories, 3 files

DOWNLOADING EXECUTABLES WITH wget ^

May be your program wants to execute some script that comes with one of the used Perl Modules. By default, the PPMDF generated by pminstalled.pl does not generate information about the scripts used by a module.

As an example of this kind of scenario let us consider the following (trivial/non sense) example:

  p2@nereida:~/LRemoteUse/examples$ cat usinganexecutable.pl
  #!/usr/bin/perl -I../lib -w
  use strict;
  use Remote::Use config => 'wgetwithbinconfig';
  use Parse::Eyapp;
  use Parse::Eyapp::Treeregexp;

  system('eyapp -h');

The distribution of Parse::Eyapp provides two executables eyapp (the yacc-like grammar compiler) and treereg (the compiler for the tree transformation language).

Of course the first thing is to have the executables eyapp and treereg in the published place in server machine (orion in our examples). Observe the directory bin of the ~/public_html/cpan directory containing the two executables:

  casiano@orion:~/public_html/cpan$ tree -a
  .
  |-- .orion.via.web
  |-- Math
  |   `-- Prime
  |       `-- XS.pm
  |-- Parse
  |   |-- Eyapp
  |   |   |-- Base.pm
  |   |   |-- Driver.pm
  |   |   |-- Grammar.pm
  |   |   |-- Lalr.pm
  |   |   |-- Node.pm
  |   |   |-- Options.pm
  |   |   |-- Output.pm
  |   |   |-- Parse.pm
  |   |   |-- Scope.pm
  |   |   |-- Treeregexp.pm
  |   |   |-- YATW.pm
  |   |   `-- _TreeregexpSupport.pm
  |   `-- Eyapp.pm
  |-- Tintin
  |   `-- Trivial.pm
  |-- Trivial.pm
  |-- auto
  |   `-- Math
  |       `-- Prime
  |           `-- XS
  |               |-- XS.bs
  |               `-- XS.so
  `-- bin
      |-- eyapp
      `-- treereg

Remote::Use downloads whatever is specified in the associated PPMDF file. By default, pminstall.pl does not include the executable scripts associated with a module in the PPMDF file. Thus, we edit the PPMDF file generated by pminstall.pl and insert line 8:

  pp2@nereida:~/LRemoteUse/examples$ head /tmp/perl5lib/.orion.via.web | cat -n
     1  (
     2  'Trivial.pm' => { dir => '', files => [
     3          '/Trivial.pm' ] },
     4  'Tintin/Trivial.pm' => { dir => '', files => [
     5          '/Tintin/Trivial.pm' ] },
     6  'Parse/Eyapp.pm' => { dir => '',
     7    files => [ '/Parse/Eyapp.pm' ],
     8    bin => [ '/bin/eyapp', '/bin/treereg' ]
     9  },
    10  'Parse/Eyapp/Lalr.pm' => { dir => '', files => [

In any entry for a module like Some/Module.pm we can add couples with the syntax

        tag => [ 'd1/f1', 'd2/f2', ... ]

to the hash entry. The tag is arbitrary and defines a family of files related with the module. A typical entry may have the form:

    'Module/Something.pm' => { 
         dir   => '/some/path',
         files => [ 'file1', 'file2', ... ],
         bin   => [ 'binexec1', 'binexec2', ... ]
         man   => [ 'manfile1', 'manfile2', ... ]
       },

While the dir and files tags are compulsory, the others are optional. The behavior of Remote::Use for a family tag like

        tag => [ 'd1/f1', 'd2/f2', ... ]

is as follows: the family of files 'd1/f1', 'd2/f2', etc. associated with the tag will be by default downloaded to 'prefix/tag/f1', 'prefix/tag/f2', etc. Where prefix is the directory specified in the prefix option of getarg inside the configuration package. See the listing of the configuration package wgetwithbinconfig:

  pp2@nereida:~/LRemoteUse/examples$ cat wgetwithbinconfig
  package wgetwithbinconfig;
  use strict;
  use warnings;

  sub getarg {
    return (
      command => 'wget -o /tmp/wget.log',
      commandoptions => '-O',
      host => 'http://orion.pcg.ull.es/~casiano/cpan',
      prefix => '/tmp/perl5lib/',
      ppmdf => '/tmp/perl5lib/.orion.via.web',
    );
  }

  sub postbin {
    my $class = shift;

    chmod 0755, @_;
  }

  1;

Thus, when Remote::Use sees the line 8 of /tmp/perl5lib/.orion.via.web (see above the full contents of the file):

   8    bin => [ '/bin/eyapp', '/bin/treereg' ]

it will transfer the file http://orion.pcg.ull.es/~casiano/cpan/bin/eyapp to the file /tmp/perl5lib/bin/eyapp in the local machine.

There is an additional problem that does not occur when using rsync for the file transference. While the transference with rsync usually preserves the permisions, the files transferred with wget do not have the execution bit set. This is the reason to define the hook postbin in the configuration package:

  sub postbin {
    my $class = shift;

    chmod 0755, @_;
  }

If a subroutine with name posttag exists in the configuration package it will be executed for each file specified in the tag family just after the file was downloaded. The posttag subroutine receives as arguments the configuration package name and the name of the downloaded file (i.e. wgetwithbinconfig and /tmp/perl5lib/bin/eyapp in the example).

Of course, to be sure that the new executables and that the just downloaded libraries will be found by the client script usinganexecutable.pl we have to properly set the PATH and PERL5LIB environment variables:

  pp2@nereida:~/LRemoteUse/examples$ export PATH=${PATH}:/tmp/perl5lib/bin/
  pp2@nereida:~/LRemoteUse/examples$ export PERL5LIB=/tmp/perl5lib/

Now we can succesfully execute the script that makes use of the remote scripts:

  pp2@nereida:~/LRemoteUse/examples$ time usinganexecutable.pl

  Usage:  eyapp [options] grammar[.yp]
    or    eyapp -V
    or    eyapp -h

      -m module   Give your parser module the name <module>
                  default is <grammar>
      -v          Create a file <grammar>.output describing your parser
      -s          Create a standalone module in which the driver is included
      -n          Disable source file line numbering embedded in your parser
      -o outfile  Create the file <outfile> for your parser module
                  Default is <grammar>.pm or, if -m A::Module::Name is
                  specified, Name.pm
      -t filename Uses the file <filename> as a template for creating the parser
                  module file.  Default is to use internal template defined
                  in Parse::Eyapp::Output
      -b shebang  Adds '#!<shebang>' as the very first line of the output file

      grammar     The grammar file. If no suffix is given, and the file
                  does not exists, .yp is added

      -V          Display current version of Parse::Eyapp and gracefully exits
      -h          Display this help screen


  real    0m0.358s
  user    0m0.224s
  sys     0m0.100s

As a consequence of the execution the directory bin/ and the executables have been added to the prefix directory:

  pp2@nereida:~/LRemoteUse/examples$ tree /tmp/perl5lib/
  /tmp/perl5lib/
  |-- bin
  |   |-- eyapp
  |   `-- treereg
  `-- files
      |-- Math
      |   `-- Prime
      |       `-- XS.pm
      |-- Parse
      |   |-- Eyapp
      |   |   |-- Base.pm
      |   |   |-- Driver.pm
      |   |   |-- Grammar.pm
      |   |   |-- Lalr.pm
      |   |   |-- Node.pm
      |   |   |-- Options.pm
      |   |   |-- Output.pm
      |   |   |-- Parse.pm
      |   |   |-- Treeregexp.pm
      |   |   `-- YATW.pm
      |   `-- Eyapp.pm
      `-- auto
          `-- Math
              `-- Prime
                  `-- XS
                      |-- XS.bs
                      `-- XS.so

THE pretag METHOD ^

Adding executables when using rsync is much simpler since rsync preserves the attributes. Just change the configuration file in the former example:

  pp2@nereida:~/LRemoteUse/examples$ cat -n usinganexecutablewithrsync.pl
     1  #!/usr/bin/perl -w
     2  use strict;
     3  use Remote::Use config => 'rsyncconfigsilent';
     4  use Parse::Eyapp;
     5  use Parse::Eyapp::Treeregexp;
     6
     7  system('eyapp -h');

The configuration file is similar to the presented in the "USING REMOTE MODULES WITH rsync VIA SSH" section.

  pp2@nereida:~/LRemoteUse/examples$ cat -n rsyncconfigsilent
     1  package rsyncconfigsilent;
     2
     3  sub getarg {
     4    my ($class, $self) = @_;
     5
     6    return (
     7      host => 'orion:',
     8      prefix => '/tmp/perl5lib/',
     9      command => 'rsync -aue ssh',
    10      ppmdf => '/tmp/perl5lib/.orion.installed.modules',
    11    );
    12  }
    13
    14  # Store executable in current directory
    15  sub prebin {
    16    my ($package, $url, $file, $self) = @_;
    17
    18    # Remove path
    19    print "downloading $url. Default place: $file ";
    20    $file =~ s{.*/}{};
    21    print "final place: $file\n";
    22    $file;
    23  }
    24
    25  1;

If for a given tag a subroutine with name pretag exists in the configuration package it will be executed for each file specified in the tag family just before the file is downloaded. The pretag subroutine receives as arguments the configuration package name, the full description of the file to download in the server (something like orion:/usr/local/bin/eyapp), the default name of the file in the client (i.e. something like /tmp/perl5lib/bin/eyapp) and a reference to the Remote::Use object. It must return the definitive full name of the file in the client (i.e. something like /home/mylogin/bin/eyapp).

For this example to work, we have to edit the PPMDF file and add the bin entry to the descriptor of Parse::Eyapp. Notice line 19:

  pp2@nereida:~/LRemoteUse/examples$ head -20 /tmp/perl5lib/.orion.installed.modules | cat -n
     1  (
     2  'CPAN/Config.pm' => { dir => '/etc/perl', files => [
     3          '/etc/perl/CPAN/Config.pm' ] },
     4  'IO/Tty.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
     5          '/usr/local/lib/perl/5.8.8/auto/IO/Tty/Tty.so',
     6          '/usr/local/lib/perl/5.8.8/auto/IO/Tty/Tty.bs',
     7          '/usr/local/lib/perl/5.8.8/IO/Tty.pm' ] },
     8  'IO/Pty.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
     9          '/usr/local/lib/perl/5.8.8/IO/Pty.pm' ] },
    10  'IO/Tty/Constant.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
    11          '/usr/local/lib/perl/5.8.8/IO/Tty/Constant.pm' ] },
    12  'Math/Prime/XS.pm' => { dir => '/usr/local/lib/perl/5.8.8', files => [
    13          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.so',
    14          '/usr/local/lib/perl/5.8.8/auto/Math/Prime/XS/XS.bs',
    15          '/usr/local/lib/perl/5.8.8/Math/Prime/XS.pm' ] },
    16  'Parse/Eyapp.pm' => {
    17     dir => '/usr/local/share/perl/5.8.8',
    18     files => [ '/usr/local/share/perl/5.8.8/Parse/Eyapp.pm' ],
    19     bin => [ '/usr/local/bin/eyapp', '/usr/local/bin/treereg' ]
    20   },

When the script is executed for the first time we get an output like:

  pp2@nereida:~/LRemoteUse/examples$ usinganexecutablewithrsync.pl
  downloading orion:/usr/local/bin/eyapp. Default place: /tmp/perl5lib//bin/eyapp final place: eyapp
  downloading orion:/usr/local/bin/treereg. Default place: /tmp/perl5lib//bin/treereg final place: treereg

  ....

Now the executables are available in the current directory:

  pp2@nereida:~/LRemoteUse/examples$ ls -ltr eyapp treereg
  -r-xr-xr-x 1 pp2 pp2 9984 2007-11-02 12:40 treereg
  -r-xr-xr-x 1 pp2 pp2 7102 2007-11-02 12:40 eyapp

APPENDIX: AUTOMATIC AUTHENTICATION ^

SSH includes the ability to authenticate users using public keys. Instead of authenticating the user with a password, the SSH server on the remote machine will verify a challenge signed by the user's private key against its copy of the user's public key. To achieve this automatic ssh-authentication you have to:

ACKNOWLEDGMENTS ^

This work has been supported by CEE (FEDER) and the Spanish Ministry of Educacion y Ciencia through Plan Nacional I+D+I number TIN2005-08818-C04-04 (ULL::OPLINK project http://www.oplink.ull.es/). The University of La Laguna has also supported my work in many ways and for many years.

Finally, thanks to Juana, Coro and my students at La Laguna.

SEE ALSO ^

AUTHOR ^

Casiano Rodriguez Leon (casiano.rodriguez.leon at gmail dot com)

LICENCE AND COPYRIGHT ^

Copyright (c) 2007 Casiano Rodriguez-Leon (casiano.rodriguez.leon at gmail dot com). All rights reserved.

These modules are free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

syntax highlighting:
79347 Uploads, 24934 Distributions 107125 Modules, 9712 Uploaders
hosted by YellowBot
do. tag. write. share.