WebmasterWorld Bans Search Engine Bots from Crawling

Nov 21, 2005 - 11:19 am 2 — by
Filed Under SEO Forum News

Danny reports that WebmasterWorld Bans Spiders From Crawling where he points to a thread Brett started which he named lets try this for a month or three...

Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; "We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots - but it is becoming an increasingly difficult problem to control."

Members are worried about how they can search WebmasterWorld, since the internal search engine is sub-par. Brett hopes that they can allow the bots again within 60 days, but it depends. Best of luck Brett!

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Legal Search Issues

Google Loses Another Round in Legal Battle Against SerpApi Over Scraping

Nov 10, 2026 - 7:09 pm
ChatGPT

Businesses in ChatGPT Business Hub Live With Click Data

Nov 10, 2026 - 1:45 pm
Google Updates

Google November 2026 Core Update Is Rolling Out

Nov 10, 2026 - 1:25 pm
Google

Gemini 4 Carbon Available In Google Search AI Mode

Nov 10, 2026 - 11:50 am
Yahoo Search Engine

Yahoo Search Launches Pivot Search, Yahoo Scout App & More

Nov 10, 2026 - 10:50 am
Search Forum Recap

Daily Search Forum Recap: November 10, 2026

Nov 10, 2026 - 10:00 am
Previous Story: AdSense Turns More AdBright Like with Advertise Here Links