Alain Barbet > Search-Circa-1.18 > Circa::Indexer


Annotate this POD

View/Report Bugs


Circa::Indexer - provide functions to administrate Circa, a www search engine running with Mysql


 use Circa::Indexer;
 my $indexor = new Circa::Indexer;
 die "Erreur à la connection MySQL:$DBI::errstr\n"
   if (!$indexor->connect);
 $indexor->addSite({url   => "",
                    email => '',
                    title => "Alian Web Server"});
 my ($nbIndexe,$nbAjoute,$nbWords,$nbWordsGood) = $indexor->parse_new_url(1);
 print   "$nbIndexe pages indexées,"
   "$nbAjoute pages ajoutées,"
   "$nbWordsGood mots indexés,"
   "$nbWords mots lus\n";

Look too in circa_admin,admin.cgi,admin_compte.cgi


This is Circa::Indexer, a module who provide functions to administrate Circa, a www search engine running with Mysql. Circa is for your Web site, or for a list of sites. It indexes like Altavista does. It can read, add and parse all url's found in a page. It add url and word to MySQL for use it at search.

This module provide routine to :


How it's work ?

Circa parse html document. convert it to text. It count all word found and put result in hash key. In addition of that, it read title, keywords, description and add a weight to all word found.

Example: A config:

 my %ConfigMoteur=(
  'author'              => '', # Responsable du moteur
  'temporate'           => 1,  # Temporise les requetes sur le serveur de 8s.
  'facteur_keyword'     => 15, # <meta name="KeyWords"
  'facteur_description' => 10, # <meta name="description"
  'facteur_titre'       => 10, # <title></title>
  'facteur_full_text'   => 1,  # reste
  'facteur_url'         => 15, # Mots trouvés dans l'url
  'nb_min_mots'         => 2,  # facteur min pour garder un mot
  'niveau_max'          => 7,  # Niveau max à indexer
  'indexCgi'            => 0,  # Index lien des CGI (ex: ?nom=toto&riri=eieiei)

A html document:

 <meta name="KeyWords"
 <meta name="Description" 
       CONTENT="Rubriques Informatique (Internet,Java,Javascript, CGI, Perl)">
 <title>Alian Web Server:Informatique,Société,Loisirs,Voyages</title>
 different word: cgi, perl, cgi

After parsing I've a hash with that:

 $words{'informatique'}= 15 + 10 + 10 =35
 $words{'cgi'} = 15 + 10 +1
 $words{'different'} = 1

Words is add to database if total found is > $ConfigMoteur{'nb_min_mots'} (2 by default). But if you set to 1, database will grow very quicly but allow you to perform very exact search with many worlds so you can do phrase searches. But if you do that, think to take a look at size of table relation.

After page is read, it's look into html link. And so on. At each time, the level grow to one. So if < to $Config{'niveau_max'}, url is added.

Class Interface ^

Constructors and Instance Methods


You can use the following keys in PARAMHASH:


Default: '', appear in log file of web server indexed (as agent)


Default: 1, boolean. If true, wait 8s between request on same server and LWP::RobotUA will be used. Else this is LWP::UserAgent (more quick because it doesn't request and parse robots.txt rules, but less clean because a robot must always say who he is, and heavy server load is avoid).


Default: 15, weight of word found on meta KeyWords


Default:10, weight of word found on meta description"


Default:10, weight of word found on <title></title>


Default:1, weight of word found on rest of page


Default: 15, weight of word found in url


Default: 2, minimal number of times a word must be found to be added


Default: 7, Maximal number of level of links to follow


Default 0, follow of not links of CGI (ex: ?nom=toto&riri=eieiei)

size_max size

Get or set size max of file read by indexer (For avoid memory pb).

host_indexed host

Get or set the host indexed.

set_host_indexed url

Set base directory with $url. It's used for restrict access only to files found on sub-directory on this serveur.

proxy adresse proxy

Get or set proxy for LWP::Robot or LWP::Agent

Ex: $circa->proxy('');

Methods use for global adminstration

addSite ref_hash

ref_hash can have these keys: url, email, title, categorieAuto, cgi, rep, file

Create account with first url url. Return id of account created

parse_new_url id account

Parse les pages qui viennent d'être ajoutée. Le programme va analyser toutes les pages dont la colonne 'parse' est égale à 0.

Retourne le nombre de pages analysées, le nombre de page ajoutées, le nombre de mots indexés.

update nb days, id account

Update url not visited since nb days for account id account. If idp is not given, 1 will be used. Url never parsed will be indexed.

Return ($nb,$nbAjout,$nbWords,$nbWordsGood)

  • $nb: Number of links find
  • $nbAjout: Number of links added
  • $nbWords: Number of word find
  • $nbWordsGood: Number of word added

Create tables needed by Circa - Cree les tables necessaires à Circa:

  • categorie : Catégories de sites
  • links : Liste d'url
  • responsable : Lien vers personne responsable de chaque lien
  • relations : Liste des mots / id site indexes
  • inscription : Inscriptions temporaires

Drop all table in Circa ! Be careful ! - Detruit touted les tables de Circa

drop_table_circa_id id account

Drop table for account id account

create_table_circa_id id account

Create tables needed by Circa for account id account


Catégories de sites


Liste d'url


Liste des mots / id site indexes


Liste des requetes

export [mysqldump], [directory of export]

Export data from Mysql in directory of export/circa.sql with mysqldump.

mysqldump: path of bin of mysqldump. If not given, search in /usr/bin/mysqldump, /usr/local/bin/mysqldump, /opt/bin/mysqldump.

<directory of export>: path of directory where circa.sql will be created. If not given, create it in $CircaConf::export, else in /tmp directory.

import_data [path_of_bin_mysql], [path_of_circa_file]

Import data in Mysql from circa.sql

path_of_bin_mysql : path to reach bin of mysql. If not given, search in /usr/bin/mysql, /usr/local/bin/mysql, /opt/bin/mysql, ENV{PATH}

path_of_circa_file : path of directory where circa.sql will be read. If not given, read it from $CircaConf::export, else /tmp directory.

Method for administrate each account

admin_compte id account

Return hash with some informations account id account Keys are:


Email address given with account creation


Title given with account creation


Number of url for this account


Number of world stored


Date of last index process


Number of request asked


Url given with account creation

most_popular_word nb item to display, id account

Retourne la reference vers un hash representant la liste des $max mots les plus présents dans la base de reponsable $id

stat_request id account

Return some statistics about request make on Circa

inscription email, url, titre

Inscrit un site dans une table temporaire

HTML functions

header_compte CGI object, id account, url of script

Function use with CGI admin_compte.cgi. Display list of features of admin_compte.cgi for this account

get_liste_liens id account

Return a html select buffer with list of url for account id account

get_liste_liens_a_valider id account,CGI object

Return a html select buffer with link to valid for account id account

get_liste_site cgi object

Return a html select buffer with list of account

get_liste_langues id account, default value, CGI object

Return a html select buffer with distinct known languages found at index time

get_liste_mot id account, id url

Return a html buffer with words found at index time for url id url.


Search::Circa, Root class for circa

Search::Circa::Parser, Manage Parser of Indexer

circa_admin, command line to use indexer


$Revision: 1.39 $



syntax highlighting: