Scrape Bots Vs. Search Bots :: Fighting the Battle

Sep 12, 2006 - 7:06 am 1 by
Filed Under Cloaking

A Search Engine Watch Forums thread asks how can one prevent scraping of his site's content by a non-authorized spider, while not hurting his rankings in search engines?

This is a serious issue, serious enough that there was a session about this named The Bot Obedience Course at SES San Jose 2006. In that session, Bill Atchison from gave an excellent presentation.

Robert Charlton at the thread notes that Bill will be releasing a software tool that helps do just that. He said there is a "Beta version coming soon." The page has details of the technology developed by

CrawlWall uses the following technology to secure your website and protect your content. All of the various methods are designed to work together in harmony to make sure that all of the spiders with permission and legitimate visitors get into your website without issue and all of the rogue crawlers get stopped and never gain admission.

Tactics such as dynamic robots.txt files, whitelist opt-in permissions, "second pass filters," ip banning or/and address banning, proxy blocking, creating certain obstacles, and a quarantine list for those uncertain IPs.

I am looking forward to seeing how it works in the real world.

Forum discussion at Search Engine Watch Forums.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 15, 2024

Jul 15, 2024 - 10:00 am
Google Search Engine Optimization

Google To Host More Creator Meetings

Jul 15, 2024 - 7:51 am

Google Tests AI-Organized Local Search Results Page

Jul 15, 2024 - 7:41 am

Report: Google AI Overviews Barely Showing Reddit Or Quora Citations Anymore

Jul 15, 2024 - 7:31 am

Google May Be Rolling Out 3D/AR Images More Widely

Jul 15, 2024 - 7:21 am
Google Search Engine Optimization

Google: Removing Massive Disavow Lists Should Not Worry You

Jul 15, 2024 - 7:11 am
Previous Story: Search Industry Pays Respect to 9/11