By 1994 the Internet was the primary source of discussion at most computer science programs. Things were no different at the University of Washington where student Brian Pinkerton developed a small single-user application to find information on the Web. At the encouragement of fellow students, Pinkerton built a web interface to his WebCrawler program, which was released on April 20, 1994, with a database containing documents from over 6,000 web servers.

The WebCrawler was unique in that it was the first web robot to capable of indexing every word on a web page, while other bots were storing a URL, a title and at most 100 words. In a few short months, the WebCrawler was averaging 15,000 hits a day. Demand eventually crashed the University of Washington’s network. America Online (AOL) eventually bought WebCrawler and ran it on its system. Excite bought WebCrawler in 1997.

Your Options...