Verify The Bots Accessing Your Site: Is Google.com Sending That GoogleBot?

Mar 7, 2007 - 7:13 am 1 by
Filed Under Google

There is no doubt that a ton of bot activity on one's sites are from rogue spiders. Spider or bots that pretend to be legit bots but are there to steal your content. We have covered several sessions on this in the past; here are some:

A new Cre8asite Forums thread asks a question on how does one verify if GoogleBot is really from Google.

Matt Cutts posted a detailed How to verify Googlebot back at the Webmaster Central Blog on 9/20/2006 explaining how to do reverse DNS and then a forward DNS->IP lookup.

Telling webmasters to use DNS to verify on a case-by-case basis seems like the best way to go. I think the recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP lookup using that googlebot.com name; eg:

> host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

> host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1

I don't think just doing a reverse DNS lookup is sufficient, because a spoofer could set up reverse DNS to point to crawl-a-b-c-d.googlebot.com.

Of course there are some ways to automate this. Either code it yourself, buy CrawlWall or implement a solution similar to Ekstreme's PHP Search Engine Bot Authentication.

Rogue spiders are no fun, as we have seen in cases with some forums.

Forum discussion at Cre8asite Forums.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 21, 2024

May 21, 2024 - 10:00 am
Google Search Engine Optimization

PSA: Site Reputation Abuse Policy Also Applies To Google Discover

May 21, 2024 - 7:51 am
Google

Google's Sundar Pichai On AI Overviews Killing Web, Smaller Publishers & Search Console Data

May 21, 2024 - 7:41 am
Google

Google Adds Visuals To About Source Knowledge Panels

May 21, 2024 - 7:31 am
Google

Google AI Overviews Colors Do Not Indicate Confidence Of Response

May 21, 2024 - 7:21 am
Google

Google Rolling Out More Graphical & Richer Search Results

May 21, 2024 - 7:11 am
Previous Story: Do MSN Live.com Search Reinclusion Requests Work?