The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
MIKEDLR / Link_Controller-0.037 / docs / HTML / link-controller_19.html 
<HTML>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<!-- Created on February, 2  2002 by texi2html 1.64 -->
<!-- 
Written by: Lionel Cons <Lionel.Cons@cern.ch> (original author)
            Karl Berry  <karl@freefriends.org>
            Olaf Bachmann <obachman@mathematik.uni-kl.de>
            and many others.
Maintained by: Olaf Bachmann <obachman@mathematik.uni-kl.de>
Send bugs and suggestions to <texi2html@mathematik.uni-kl.de>
 
-->
<HEAD>
<TITLE>Using LinkController: History</TITLE>

<META NAME="description" CONTENT="Using LinkController: History">
<META NAME="keywords" CONTENT="Using LinkController: History">
<META NAME="resource-type" CONTENT="document">
<META NAME="distribution" CONTENT="global">
<META NAME="Generator" CONTENT="texi2html 1.64">

</HEAD>

<BODY LANG="" BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#800080" ALINK="#FF0000">

<A NAME="SEC34"></A>
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_18.html#SEC33"> &lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC35"> &gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_18.html#SEC33"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top"> Up </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<H1> E. History </H1>
<!--docid::SEC34::-->
<P>

LinkController was originally inspired by MOMspider and having the
MOMspider code available was very useful when starting the creation of
this kit, but, it shares almost no code with MOMspider, other than what
has comes to it from the LibWWW-Perl library.
</P><P>

Philosophically, the MOMspider heritage is obvious in the wish to handle
big jobs efficiently.  In the working practice there are far more
differences than similarities, partly caused by Perl language changes.
</P><P>

I decided to completely separate the exploration of the local
infostructure, looking for links to be checked, from the actual checking
process.  This means that checking can be spread over a large number of
days and still run efficiently. 
</P><P>

The basic aim of this link checking kit is to be able to efficiently
handle any size of link checking job.  At the bottom end we have
checking new pages as they are written.  Here we want to use
information from previous checks to avoid having to check all of each
page every time.  At the other end we have massive info structures
(sites) which deal in many thousands of links and could not possibly
all be checked in one day.  For this latter case the aim is to be able
to efficiently spread the link checking load into all available low
usage periods.
</P><P>

My primary aim in writing this was not to write very efficient code for
the small scale case (takes minimum time to do everything), but rather
code which would scale well.  If your system can check 1000 links in two
days, it will hopefully be able to check almost 7000 links in two weeks.
I'm trying to make sure all data structures which grow with the number
of links are kept on disk.
</P><P>

<BLOCKQUOTE><TABLE BORDER=0 CELLSPACING=0> 
<TR><TD ALIGN="left" VALIGN="TOP"><A HREF="link-controller_19.html#SEC35">E.1 Acknowledgements</A></TD><TD>&nbsp;&nbsp;</TD><TD ALIGN="left" VALIGN="TOP">People and or institutions who helped</TD></TR>
</TABLE></BLOCKQUOTE>
<P>

<A NAME="Acknowledgements"></A>
<HR SIZE="6">
<A NAME="SEC35"></A>
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC36"> &gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> Up </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<H2> E.1 Acknowledgements </H2>
<!--docid::SEC35::-->
<P>

Although I wrote this system by myself, this would not have been nearly
as easy and almost certainly wouldn't have ever been finished without
the help of the following people and organisations.
</P><P>

<BLOCKQUOTE><TABLE BORDER=0 CELLSPACING=0> 
<TR><TD ALIGN="left" VALIGN="TOP"><A HREF="link-controller_19.html#SEC36">Esoterica Internet Portugal</A></TD><TD>&nbsp;&nbsp;</TD><TD ALIGN="left" VALIGN="TOP"></TD></TR>
<TR><TD ALIGN="left" VALIGN="TOP"><A HREF="link-controller_19.html#SEC37">IPPT PAN Poland</A></TD><TD>&nbsp;&nbsp;</TD><TD ALIGN="left" VALIGN="TOP"></TD></TR>
<TR><TD ALIGN="left" VALIGN="TOP"><A HREF="link-controller_19.html#SEC38">The Tardis Project</A></TD><TD>&nbsp;&nbsp;</TD><TD ALIGN="left" VALIGN="TOP"></TD></TR>
<TR><TD ALIGN="left" VALIGN="TOP"><A HREF="link-controller_19.html#SEC39">Other Free Software Authors</A></TD><TD>&nbsp;&nbsp;</TD><TD ALIGN="left" VALIGN="TOP"></TD></TR>
</TABLE></BLOCKQUOTE>
<P>

<A NAME="Esoterica Internet Portugal"></A>
<HR SIZE="6">
<A NAME="SEC36"></A>
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC35"> &lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC37"> &gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC35"> Up </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<H3> Esoterica Internet Portugal </H3>
<!--docid::SEC36::-->
<P>

Esoterica provided me with full access to the Internet in Portugal and
use of their computers for free which allowed me to keep up on both this
software and the Linux Access HOWTO.  In particular I'd like to thank
all of the members of staff who helped me very much.  These people include
Mario Francisco Valente (the instigator of Mini Linux) who first
agreed to me using their kit, set me up to use their machines, and along
with 
Luis Sequeira provided a sounding board for some ideas.
Luis also provided the odd lift home in the evening.  Also 
Martim de Magalhaes Pereira and Mr Mendes.  See them all on 
</P><P>

<A HREF="http://www.esoterica.pt/esoterica/quemsomos.html">http://www.esoterica.pt/esoterica/quemsomos.html</A>
</P><P>

For more about esoterica (Internet Services in Portugal) see:
</P><P>

<A HREF="http://www.esoterica.pt/esoterica/">http://www.esoterica.pt/esoterica/</A>
</P><P>

These pages are in Portugese<A NAME="DOCF3" HREF="link-controller_fot.html#FOOT3">(3)</A> of course.
</P><P>

<A NAME="IPPT PAN Poland"></A>
<HR SIZE="6">
<A NAME="SEC37"></A>
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC36"> &lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC38"> &gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC35"> Up </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<H3> IPPT PAN Poland </H3>
<!--docid::SEC37::-->
<P>

Thanks go to IPPT PAN (part of PAN - Polska Akademia Naukowa) in Poland
and in particular Piotr Pogorzelski who allowed me use of facilities for
testing this software, provided a willing victim for having his web
pages tested and made a number of suggestions which have been
incorporated into the software.
</P><P>

<A NAME="The Tardis Project"></A>
<HR SIZE="6">
<A NAME="SEC38"></A>
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC37"> &lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC39"> &gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC35"> Up </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<H3> The Tardis Project </H3>
<!--docid::SEC38::-->
<P>

Supported by the Computing Science department of the University of
Edinburgh, the Tardis project provides an experimental framework in
which students, former students and other related people to do their own
work on fully Internet connected Unix and Linux hosts.
</P><P>

The use of the facilities of the Tardis Project has made it much easier
for me to develop software like this.  In particular, the large amount
of disk space the administrators have allow me to use is very useful.
</P><P>

<A NAME="Other Free Software Authors"></A>
<HR SIZE="6">
<A NAME="SEC39"></A>
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC38"> &lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC35"> Up </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<H3> Other Free Software Authors </H3>
<!--docid::SEC39::-->
<P>

It is through the software provided by the Free Software Foundation
(such as the <CODE>gcc</CODE> C compiler, Emacs, the file utilities), the
authors of the various packages which make up a working Linux System
(Linux by Linus Torvalds, Alan Cox, etc.... filesystems and support by
Theodore Tytso, Stefan Tweedie etc.. Linux-Libc by HJ Lu, based on GNU
<CODE>glibc</CODE> from the FSF.. the list is indefinite) and the authors
of Perl and its modules, especially Gisle Aas and Martijn Kostler for
LibWWW-Perl that I was able to set this up.
</P><P>

I'd particularly like to thank Tim Goodwin the author of the Perl CDB
module who made and accepted a number of alterations to that, at my
request.  These alterations made this package simpler to write and
easier to maintain.
</P><P>

The Free Software Foundation web pages are at
</P><P>

<A HREF="http://www.gnu.ai.mit.edu/">http://www.gnu.ai.mit.edu/</A>
</P><P>

<A NAME="Invoking the Programs"></A>
<HR SIZE="6">
<TABLE CELLPADDING=1 CELLSPACING=1 BORDER=0>
<TR><TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_19.html#SEC34"> &lt;&lt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_20.html#SEC40"> &gt;&gt; </A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT"> &nbsp; <TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller.html#SEC_Top">Top</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_toc.html#SEC_Contents">Contents</A>]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[Index]</TD>
<TD VALIGN="MIDDLE" ALIGN="LEFT">[<A HREF="link-controller_abt.html#SEC_About"> ? </A>]</TD>
</TR></TABLE>
<BR>  
<FONT SIZE="-1">
This document was generated
by <I>Michael De La Rue</I> on <I>February, 2  2002</I>
using <A HREF="http://www.mathematik.uni-kl.de/~obachman/Texi2html
"><I>texi2html</I></A>

</BODY>
</HTML>