WebmasterWorld Bans Search Engine Bots from Crawling

Nov 21, 2005 - 11:19 am 2 by
Filed Under SEO Forum News

Danny reports that WebmasterWorld Bans Spiders From Crawling where he points to a thread Brett started which he named lets try this for a month or three...

Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; "We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots - but it is becoming an increasingly difficult problem to control."

Members are worried about how they can search WebmasterWorld, since the internal search engine is sub-par. Brett hopes that they can allow the bots again within 60 days, but it depends. Best of luck Brett!

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: August 28, 2025

Aug 28, 2025 - 10:00 am
Google Search Engine Optimization

Some Sites Have Huge Crawl Rate Declines In Google Search Console

Aug 28, 2025 - 9:30 am
Google Search Engine Optimization

Bug: Google Search Console Performance Report Search Appearance Android App

Aug 28, 2025 - 7:51 am
Bing Ads

Microsoft Testing Tests Larger Titles For Ads In Bing Search

Aug 28, 2025 - 7:41 am
Google Ads

Google Ads Gets Audience Exclusions For Shopping Campaigns

Aug 28, 2025 - 7:31 am
Bing Search

Bing Tests Bolding Site Name On Hover

Aug 28, 2025 - 7:21 am
 
Previous Story: AdSense Turns More AdBright Like with Advertise Here Links