Salient Marketing  - search engine marketing, search engine optimization, market research and market segmentation
HomeOur CompanyMarketing ServicesMarketing PartnersMedia RoomMarketing ResourcesContact Us
Search Engine Optimization Resources

The first web robot - 1993
home > resources > history of search engines

The first web robot was the creation of Massachusetts Institute of Technology (MIT) physics student Matthew Gray in 1993. Gray’s World Wide Web Wanderer was designed to track the growth of the then-infant Web.

“I wrote the Wanderer to systematically traverse the Web and collect sites,” Gray wrote of his invention. “I was initially motivated primarily to discover new sites, as the Web was still a relatively small place. The Wanderer was the primary tool for collection of data to measure the growth of the Web. It was the first automated Web agent or “spider.” The Wanderer was first functional in spring of 1993 and performed regular traversals of the Web from June 1993 to January 1996.”

During its three-year run, the Wanderer tracked the growth in web sites from 130 in June 1993, to more than 100,000 in January 1996 and an estimated 230,000 just six months later.

Gray extended the scope of the Wanderer from tracking the Web’s size to capturing individual URLs into Wandex, the first web database. Gray’s good intentions also created controversy as early versions of the Wanderer were also known to not just crawl the Web, but slow traffic on the Web to a crawl as the program repeatedly accessed the same pages hundreds of times a day. The problem was fixed in later versions.

Here is a growing archive of the search engine history:

  • 2003 – Microsoft redux
  • 2001 – Goodbye GoTo, hello Overture
  • 2000 – Some Web expertise
  • 1999 – A winning search concept
  • 1999 – Now that’s fast!
  • 1998 – Its a hit! DirectHit
  • 1998 – And in the other corner... MSN
  • 1998 – And in this corner… Google
  • 1998 – The Open (source) Directory Project
  • 1997 – Ask Jeeves, the butler did it
  • 1997 – GoTo, What do I bid?
  • 1996 – LookSmart – the Australian connection
  • 1996 – HotBot, one hot bot!
  • 1995/96 – The Northern Light
  • 1995 – AltaVista
  • 1995 – The Meta-search
  • 1994-95 – InfoSeek
  • 1994 – Lycos
  • 1994 – The WebCrawler
  • 1994 – Yahoo!
  • 1994 – A Galaxy of web pages
  • 1993 – The birth of Excite
  • 1993 - The Web’s oldest existing search engine
  • 1993 – The first web robot
  • 1993 - Archie’s pal, Jughead
  • 1993 - Veronica, the grandmother of search engines
  • 1989 - Archie, the grandfather of all search engines

SalientMarketing is Search Engine Marketing and Market Research Firm. Our staff of marketing and technology professionals deliver services that maximize the return on investment (ROI) for online and offline marketing initiatives. Our process of implementation is rooted in upfront research followed by a measurement of success, ensuring that our strategies are on target and exceed expectations. It's all about taking the guesswork of business success.

>> Click here for our list of services or solutions that we provide

SEO Resources Menu
» Search Engine 101
» What is a search engine?
» Search Engine Positioning
» Introduction to SEP
» Benefits of SEP
» Components of Search Engine Positioning
» Return of Investment (ROI)
» Search Engine Marketing
» Search Engine Review
Search engine marketng strategies that affect your bottom line

Home | Company | Services | Partners | Media Room | Resources | Contact Us | Site Map | Client Login

2006© Salient Marketing - search engine market research, market segmentation,
search engine positioning, web personalization and technology/web applications