CPAN::Search::Lite::Index - set up or update database tables.
my $index = CPAN::Search::Lite::Index->new(config => 'cpan.conf', setup => 1); $index->index();
This is the main module used to set up or update the database tables used to store information from the CPAN and ppm indices. The creation of the object
my $index = CPAN::Search::Lite::Index->new(%args);
accepts three arguments:
config => /path/to/config.conf
This argument specifies where to find the configuration file used to determine the remaining options. In lieu of this option, the environment variable CSL_CONFIG_FILE pointing to the configuration file may be specified.
CSL_CONFIG_FILE
setup => 1
This (optional) argument specifies that the database is being set up. Any existing tables will be dropped.
reindex => value
This (optional) argument specifies distribution names that one would like to reindex in an existing database. These may be specified as either a scalar, for a single distribution, or as an array reference for a list of distributions.
Most of the options used to control the behaviour of the indexing are contained in a configuration file. An example of the format of such a file is
[CPAN] CPAN = /var/ftp/pub/CPAN pod_root = /usr/local/POD html_root = /usr/local/httpd/htdocs/CPAN [DB] db = pause user = sarah passwd = lianne [WWW] css = cpan.css up_img = up.gif tt2 = /usr/local/tt2 geoip = /usr/local/share/geoip/cpan.txt
This consists of 3 sections.
This is associated with various things related to CPAN.
CPAN = /var/ftp/pub/CPAN
This specifies the root directory of the local CPAN mirror, if this exists, or the location where the CPAN index files will be downloaded and kept, if the no_mirror option is specified.
no_mirror
pod_root = /usr/local/POD
This specifies where the extracted pod files from a distribution will be kept. A subdirectory dist_name under this directory will be created corresponding to the name of the distribution.
dist_name
pod_only = 1
This specifies that, if the module files are to be extracted, fetch only those that contain pod.
split_pod = 1
This specifies that, if the module files are to be extracted, when generating the html pages create two pages for each module: one containing just the documentation, and the other containing the code run through Perl::Tidy. For a module such as Foo::Bar, the documentation will be saved as a file Foo/Bar.html, while the sources will be saved as Foo/Bar.pm.html.
Perl::Tidy
Foo::Bar
html_root = /usr/local/httpd/htdocs/CPAN
This specifies where the html files created from the pod files will be kept. A subdirectory dist_name under this directory will be created corresponding to the name of the distribution.
ignore = some_dist_name_to_ignore
This specifies a name of a distribution (without a version number) to ignore in indexing. This option may be given a number of times to specify an array of values, or may be specified as
ignore = <<EOL Module-CPANTS-asHash CORBA-IDL EOL
This array of values (which may include regular expressions) is joined together as
$pat = join '|', @ignore_dists
and if the distribution name matches
$dist_name =~ /^($pat)$/
the distribution is ignored.
no_mirror = 1
This specifies that a local CPAN mirror isn't available, and as such no pod or html files will be extracted nor created.
no_ppm = 1
This can be used to signal to not gather information on Win32 ppm packages from the repositories specified in $repositories of CPAN::Search::Lite::Util.
$repositories
remote_mirror = http://cpan.wherever.edu
If no_mirror is specified, the value of remote_mirror will be used to fetch the CPAN indices. If not given, http://www.cpan.org will be used.
remote_mirror
multiplexer = http://cpan.redirect.edu/cpan
This can be used to specify a multiplexer to redirect downloads to nearby CPAN mirrors. See, for example, Apache::GeoIP for one implementation.
cat_threshold = 0.99
Many modules do not have a category (chapter) associated with them. In such cases, when populating the database, the AI::Catgorizer module is used to guess which category should be assigned to such a module, based on available information for those modules that do have a category. The value of cat_threshold is used to determine if the guessed category should be accepted (a perfect match has a score of 1, and no match has 0). If no such value is given, a default of 0.995 is used.
no_cat = 1
Set no_cat equal to a true value if you don't want AI::Categorizer to try categorizing modules which don't have a category assigned.
This is used to store connection information to the database used to populate the tables.
db = pause
This is the name of the database used. It is assumed here that the database has already been created, and that appropriate read, write, update, create, and delete permissions for the user specified below have already been granted.
user = sarah
This is the user under which to connect to the database.
passwd = lianne
This is the password to use for the user.
This is used for various information related to a web interface.
css = cpan.css
If specified, this will be used as the css file when generating the html files from the pod files of the distribution. It is assumed this file appears directly beneath the html_root of the CPAN section.
html_root
CPAN
up_img = up.gif
If specified, this will be used as an image in the generated html files linking each section to the top-most index. If not specifed, the text __top will be used. It is assumed this image appears directly beneath the html_root of the CPAN section.
dist_info = http://cpan.uwinnipeg.ca/dist/
If specified, this will be used to provide a link on the generated html pages to information on the distribution. The name of the distribution will be added at the end of the link (for example, using http://cpan.uwinnipeg.ca/dist/ will result, for the libnet distribution, in a link to http://cpan.uwinnipeg.ca/dist/libnet.
tt2 = /usr/local/tt2
This gives the location of the Template-Toolkit pages used to provide a web interface. This is used to place a file mirror_list (extracted from $CPAN/MIRRORED.BY) containing a list of CPAN mirrors.
geoip = /usr/local/share/geoip/cpan.txt
If the module Geo::IP or Apache::GeoIP is used to provide a redirection service to a nearby CPAN mirror based on the location of origin, this file will be created to provide the necessary country of origins of the CPAN mirrors.
Geo::IP
Apache::GeoIP
Calling
$index->index();
will start the indexing procedure. Various messages detailing the progress will written to STDOUT, which by default will be captured into a file cpan_search_log.dddddddddd, where the extension is the time that the method was invoked. Passing index an argument of log => log_file will save these messages into log_file. Error messages are not captured, and will appear in STDERR.
time
index
log => log_file
The steps of the indexing procedure are as follows.
fetch index data
If the no_mirror option is specified, the necessary CPAN index files $CPAN/MIRRORED.BY, $CPAN/indices/ls-lR.gz, $CPAN/authors/01mailrc.txt.gz, $CPAN/modules/02packages.details.txt.gz, and $CPAN/modules/03modlist.data.gz will be fetched from the CPAN mirror specified by the $cpan variable at the beginning of CPAN::Search::Lite::Index. If you are using this option, it is recommended to use the same CPAN mirror with subsequent updates, to ensure consistency of the database. As well, the information on the locations of the CPAN mirrors used for Template-Toolkit and GeoIP is written.
$cpan
get index information
Information from the CPAN indices and, if desired, the ppm repositories is extracted. This is done through CPAN::Search::Lite::Info (for the CPAN indices) and CPAN::Search::Lite::PPM (for the ppm repositories).
get state information
Unless the setup argument within the new method of CPAN::Search::Lite::Index is specified, this will get information on the state of the database through CPAN::Search::Lite::State. A comparision is then made between this information and that gathered from the CPAN indices, and if there's a discrepency in some items, those items are marked for either insertion, updating, or deletion, as appropriate.
setup
new
extract files
Unless the no_mirror option is specified, this will extract, through CPAN::Search::Lite::Extract, the available pod sections of files of a distribution, placing them under a subdirectory dist_name (corrsponding to the name of the distribution) under the specified pod_root in the configuration file. pod2html is then run on them, with the results placed under dist_name of html_root. Also, a README, Changes, INSTALL, and META.yml file, if present, will be copied over into dist_name under pod_root. Finally, information on the prerequisites of the distribution, and distribution and module descriptions, if available and needed, is extracted.
pod_root
pod2html
populate the database
At this stage the gathered information is used to populate the database, through CPAN::Search::Lite::Populate, either inserting new items, updating existing ones, or deleting obsolete items.
As well, unless the no_mirror option is specified, the html files created under html_root will be edited to adjust the links to module files. This is necessary because when the html files are created no cache is used (in order to maintain consistency between updates), and consequently links to packages outside of a given package may be incorrect. This is fixed by querying the database to see what module documentation is actually present, and then adjusting the links in the html files accordingly (or removing a link if the indicated documentation is missing).
CPAN::Search::Lite::Info, CPAN::Search::Lite::PPM, CPAN::Search::Lite::State, CPAN::Search::Lite::Extract, CPAN::Search::Lite::Populate, and CPAN::Search::Lite::Util. Development takes place on the CPAN-Search-Lite project at http://sourceforge.net/projects/cpan-search/.
This software is copyright 2004 by Randy Kobes <randy@theoryx5.uwinnipeg.ca>. Use and redistribution are under the same terms as Perl itself.
To install CPAN::Search::Lite, copy and paste the appropriate command in to your terminal.
cpanm
cpanm CPAN::Search::Lite
CPAN shell
perl -MCPAN -e shell install CPAN::Search::Lite
For more information on module installation, please visit the detailed CPAN module installation guide.