WebmasterWorld Bans Search Engine Bots from Crawling

Nov 21, 2005 - 11:19 am 2 by
Filed Under SEO Forum News

Danny reports that WebmasterWorld Bans Spiders From Crawling where he points to a thread Brett started which he named lets try this for a month or three...

Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; "We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots - but it is becoming an increasingly difficult problem to control."

Members are worried about how they can search WebmasterWorld, since the internal search engine is sub-par. Brett hopes that they can allow the bots again within 60 days, but it depends. Best of luck Brett!

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 19, 2025

May 19, 2025 - 10:00 am
Google

Google Begins Rolling Out Discover On Desktop

May 19, 2025 - 7:51 am
Google Search Engine Optimization

Google Merchant Center Now Recommends Product's Brand Name In Title

May 19, 2025 - 7:41 am
Google Search Engine Optimization

Google Sitemap Error For Some International 5 Character Subdomains

May 19, 2025 - 7:31 am
Google

Google Search Begins Migrating Off Country Specific ccTLDs

May 19, 2025 - 7:21 am
Google Ads

Google Ads Spending Bug With New Customer Acquisition Since May 15

May 19, 2025 - 7:11 am
Previous Story: AdSense Turns More AdBright Like with Advertise Here Links