As the spread of the Internet and the Web took off exponentially, its use as a social magnet grew with it. People – mostly students – began to create pages of links to other web pages that interested them, so that friends and colleagues could share them easily.

David Filo and Jerry Yang were two Ph.D. candidates at Stanford University who started their guide to web sites in a campus trailer in February 1994 to keep track of their personal interests. It wasn’t long before they spent more time on the list of web site links than on their doctoral dissertations. As the lists grew, they broke them into categories, then subcategories. At this point, the web site that started out as Jerry’s Guide to the World Wide Web was renamed Yahoo! with the help of a dictionary. While some say Yahoo! stands for Yet Another Hierarchical Officious Oracle, Filo and Yang insist it was simply named after them – a couple of “yahoos” as the dictionary stated: “Rude, unsophisticated, uncouth.”

Word of Yahoo! spread quickly and in the fall of 1994, the site had its first million-hit day, representing almost 100,000 unique visitors.

Launched in January 1994, Galaxy.com was the first searchable Internet directory. Galaxy was created as part of the Einet division at the MCC Research Consortium at the University of Texas. The original initiative was to develop tools for large-scale directory services to support electronic commerce.

The ownership of Galaxy has changed hands numerous times through a series of mergers and acquisitions. Today Galaxy.com is owned by Logika Corporation, of Chicago and has a directory of more than 3.2 million listings, across more than 680,000 categories.

The popular public search engine Excite began life as a project called Architext created by six Stanford undergraduates in February 1993. Their idea was to use statistical analysis of word relationships in order to provide more efficient searches through the large amount of information on the Internet. Their project was fully funded by mid-1993. Once funding was secured, they released a version of their search software for webmasters to use on their own web sites. The software today is called Excite for Web Servers.

A decade after Aliweb (www.aliweb.com) was created in October 1993, the Web’s oldest search engine is still functioning, albeit it nowhere near the standards of its modern-day successors.

Aliweb was developed by the United Kingdom’s Martijn Koster to provide Archie-like functionality in a search engine for the web. Aliweb does not have the robot functionality of Wanderer or other search engines and relies on web masters to post indexing information on sites they want listed. Most people don’t know how to supply the special indexing file required; therefore relatively few sites are accessible through Aliweb compared to robot-based search engines.

The first web robot was the creation of Massachusetts Institute of Technology (MIT) physics student Matthew Gray in 1993. Gray’s World Wide Web Wanderer was designed to track the growth of the then-infant Web.

“I wrote the Wanderer to systematically traverse the Web and collect sites,” Gray wrote of his invention. “I was initially motivated primarily to discover new sites, as the Web was still a relatively small place. The Wanderer was the primary tool for collection of data to measure the growth of the Web. It was the first automated Web agent or “spider.” The Wanderer was first functional in spring of 1993 and performed regular traversals of the Web from June 1993 to January 1996.”

During its three-year run, the Wanderer tracked the growth in web sites from 130 in June 1993, to more than 100,000 in January 1996 and an estimated 230,000 just six months later.

Gray extended the scope of the Wanderer from tracking the Web’s size to capturing individual URLs into Wandex, the first web database. Gray’s good intentions also created controversy as early versions of the Wanderer were also known to not just crawl the Web, but slow traffic on the Web to a crawl as the program repeatedly accessed the same pages hundreds of times a day. The problem was fixed in later versions.