WebmasterWorld Bans Search Engine Bots from Crawling

Nov 21, 2005 - 11:19 am 2 by
Filed Under SEO Forum News

Danny reports that WebmasterWorld Bans Spiders From Crawling where he points to a thread Brett started which he named lets try this for a month or three...

Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; "We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots - but it is becoming an increasingly difficult problem to control."

Members are worried about how they can search WebmasterWorld, since the internal search engine is sub-par. Brett hopes that they can allow the bots again within 60 days, but it depends. Best of luck Brett!

Forum discussion at WebmasterWorld.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 12, 2024

Jun 12, 2024 - 10:00 am
Google Search Engine Optimization

I Interviewed Elizabeth Tucker, Googler Responsible For The March Core Update

Jun 12, 2024 - 7:51 am

Google Adds Return Policy Structured Data Support To Organization Level

Jun 12, 2024 - 7:41 am
Google Maps

Google Maps Tests QR Codes On Business Profiles

Jun 12, 2024 - 7:31 am
Google Ads

Google Ads Adding New Optional Verification This Month

Jun 12, 2024 - 7:21 am

Google Tests Shaded Site Name & Favicon For Search Results

Jun 12, 2024 - 7:11 am
Previous Story: AdSense Turns More AdBright Like with Advertise Here Links