Tracking Search Bot Activity


Back in the day, tracking how bots accessed your site was a bit of a crave. Now, you don't hear about it much. The old Google Analytics, aka Urchin, had a section for displaying bot activity on your site. It did this because Urchin also analyzed your log files, in addition to the method Google Analytics uses to track based on JavaScript. Since most spiders don't load JavaScript, popular analytical software, such as Google Analytics, won't track the bot activity.

To track bot activity, you need to use analytic software that analysis your log files. There are other methods, including writing your own database script to track all bot activity. Back in the day, Darrin Ward (founder of SEO Chat, who sold it years ago) created a script that looked for bot activity and stored the data in a MySQL database. I forgot the name of the software, but I bet it is still out there now or there are plenty of alternatives.

If you don't want to instal anything but you still want to track bot activity, there are ways.

In Google Webmaster Tools, you can go to the "Crawl Stats" section under "Statistics" and get data from Google on how active GoogleBot is on your site. Google will show you data and time based graphs for:

  • Number of pages crawled per day
  • Number of kilobytes downloaded per day
  • Time spent downloading a page (in milliseconds)

Here is a screen shot of our graphs:

Google Crawl Stats

If spiders are not crawling your site, you might have to worry. Otherwise, this is often a metric not discussed often by SEOs.

Forum discussion at HighRankings Forums.

Previous story: Google's Tips on Unranking For Keywords


KJ Rodgers

04/06/2009 02:34 pm

We recently received all of our stats from the Google bot. Still trying to make out what to do with it all.

James Matthews

04/06/2009 06:02 pm

I use wassup for tracking bots. It's great at IDing the bots and filtering based on them. Also these days Live's bots execute javascript so there is a way to track them


04/07/2009 11:22 am

It's important to understand which pages are being crawled and how often. This is a key metric to help understand how quickly you might see pages in search engines. And/or to trouble shoot why they are not showing up. If they're not being crawled, you know the answer...

Patrice Albertus

04/07/2009 11:52 am

That's something very interesting but not so new. Here's a filter for GA that will help you to track inside the bots activity :

No Name

04/08/2009 10:14 pm

Good clarifications thanks.

blog comments powered by Disqus