Search results for "dist:Link_Controller robots.txt"
ElephantAgent - the agent that never forgets
This is the robot agent that never forgets. One of the major advantages of the original MOMspider link checker was that it didn't need to keep checking robots.txt files every time it was started. This agent does the same by using a disk cache of host...
MIKEDLR/Link_Controller-0.037 - 09 Feb 2002 18:12:34 UTC