Have You Been Spidered Lately? - 2/4/2005
We certainly hope so. Being spidered, better yet, being spidered often, is the goal of every web site. If you're not sure what spiders are, check out our 'spider definition'.
But how do you know which, if any, spiders have visited your site? How do you distinguish regular web users from spiders? They key is a little piece of informations called 'User Agent'. Each time a web browser or spider visits a page, it identifies itself with a 'User Agent' that distinguishes it from other types of browsers and spiders. Today, most web hosting companies provide detailed and summarized reports on the different user agents that have been requesting pages from the site. Some of these reports are very advanced, as they give each user agent a nice friendly name that most people understand. For example, it's much nicer to look at 'Yahoo' than 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)'. Unfortunately, search engine spider statistics are often grouped into normal browser statistics. However, if you know what to look for, they usually stand out in the list of browsers that have been looking at your site.
Below are the most common search engines, along with the basic names of their spiders:
AltaVista - Scooter
Excite - ArchitextSpider
Google - Googlebot
Lycos - Lycos_Spider
MSN - MSNBot
WebCrawler - WebCrawler
WiseWire - WiseWire
Yahoo! - Slurp
|
|
|
About the Author: Netensify is an extremely effective internet marketing firm that works tirelessly to increase market share and profits for it's clients. Unlike most marketing firms, Netensify's staff have in-depth knowledge of all of the technical aspects of search engines, web site development, and marketing technologies, allowing them to deliver outstanding results.
<< All Articles