A decade after Aliweb (www.aliweb.com) was created in October 1993, the Web’s oldest search engine is still functioning, albeit it nowhere near the standards of its modern-day successors.

Aliweb was developed by the United Kingdom’s Martijn Koster to provide Archie-like functionality in a search engine for the web. Aliweb does not have the robot functionality of Wanderer or other search engines and relies on web masters to post indexing information on sites they want listed. Most people don’t know how to supply the special indexing file required; therefore relatively few sites are accessible through Aliweb compared to robot-based search engines.

The first web robot was the creation of Massachusetts Institute of Technology (MIT) physics student Matthew Gray in 1993. Gray’s World Wide Web Wanderer was designed to track the growth of the then-infant Web.

“I wrote the Wanderer to systematically traverse the Web and collect sites,” Gray wrote of his invention. “I was initially motivated primarily to discover new sites, as the Web was still a relatively small place. The Wanderer was the primary tool for collection of data to measure the growth of the Web. It was the first automated Web agent or “spider.” The Wanderer was first functional in spring of 1993 and performed regular traversals of the Web from June 1993 to January 1996.”

During its three-year run, the Wanderer tracked the growth in web sites from 130 in June 1993, to more than 100,000 in January 1996 and an estimated 230,000 just six months later.

Gray extended the scope of the Wanderer from tracking the Web’s size to capturing individual URLs into Wandex, the first web database. Gray’s good intentions also created controversy as early versions of the Wanderer were also known to not just crawl the Web, but slow traffic on the Web to a crawl as the program repeatedly accessed the same pages hundreds of times a day. The problem was fixed in later versions.

Shortly after Veronica’s appearance, another Gopher search tool called Jughead was developed by Rhett “Jonzy” Jones at the University of Utah Computer Centre. Despite comical appearances, Jones claimed Jughead stood for Jonzy’s Universal Gopher Hierarch Excavation and Display. Jughead had similar functionality to Veronica, but added Boolean search capabilities. However, Jughead restricted the search to individual servers.

FTP enabled Internet users to locate and access files or more correctly, folders of information located on FTP servers. Another protocol called Gopher existed to handle plain-text documents and these were stored on so-called Gopher servers. Gopher was created in 1991 by Mark McCahill and his team at the University of Minnesota and was named after the university mascot, the Golden Gopher.

The popularity of Emtage’s Archie had grown considerably by 1993, prompting Fred Barrie and Steven Foster at the University of Nevada System Computing Services Group to develop Veronica to search for Gopher files. The group dubbed Veronica – Very Easy Rodent-Oriented Netwide Index to Computerized Archives – as “the grandmother of all search engines.”

At its peak usage, Veronica searched through a database of over 5,500 Gopher servers and over 10 million Gopher “items” or documents whose titles contain your keyword. Veronica would then set up a temporary Gopher server containing the matches which you could browse. Veronica was much overloaded, often gave error messages for a keyword search then provided results for the same keyword search on a second or third try.

The Internet’s first search engine appeared in 1989 and was invented by Alan Emtage, a computer science student from Barbados studying at McGill University. Emtage dubbed his invention Archie, a contraction of the word “archives” to fit the shortened naming conventions of the UNIX operating system.

Archie was designed to provide an online index of public FTP (file transfer protocol) sites, the Internet information repositories that existed before the Web and home pages. Before Archie, the only way people could find out the existence of an FTP server was by word of mouth or to be sent an e-mail telling them where to find the information.

By 1992, Archie had catalogued over 200 public FTP sites. It is a figure that seems almost laughable by today’s standards, but a decade ago, was already beyond anyone’s wildest dreams. At its peak in 1995, 30 Archie engines crawled the Internet and had catalogued millions of pages.

While FTP continues to be a common way to share files over the Internet, Archie is no longer used.