Tag: spiders

Older Entries

Blocking Apple Devices On Your Web Site? You May Block Google Also!

Apr 16, 2014

Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site

Jan 3, 2014

Video Recap of Weekly Search Buzz :: February 1, 2013

Feb 1, 2013

DuckDuckGo Not Respecting Robots.txt Directives?

Nov 27, 2012

Google: Robots.txt Files Must Be Smaller Than 500KB

Jan 30, 2012

BingBot IP Ranges Updated

Nov 16, 2011

Blocking Google Video Bot Wont Stop Your Pages Showing In Google Video

Dec 15, 2010

Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009

MSNBot Crawl Delay Doesn't Delay

Nov 11, 2009

Bing's MSNBot Crawl Happy?

Oct 23, 2009

Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails

Sep 17, 2009

Stop Spiders From Crawling Your Site on Shabbat, Including GoogleBot

Sep 9, 2009

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009

MSNBot Clicking on Bing adCenter Search Ads?

Jul 29, 2009

New MSNBot Named adidxbot Causing Trouble

Jul 7, 2009

Tracking Search Bot Activity

Apr 6, 2009

Multiple Robots.txt Files for Single Domain

Apr 1, 2009

Live Search Begins Crawling JavaScript with MSNBot-Media

Jan 2, 2009

Did Google Stop Crawling Blogger Blogs on Personalized Domains?

Aug 7, 2008

Did Ask.com Stop or Slow Crawling the Web?

Jul 18, 2008

Google Improves Flash Indexing Again

Jul 15, 2008

Google Retraction: Blocking Regions Is Not Cloaking

Jul 3, 2008

Updated: Google Says Blocking Countries Outside of the US is Against Policies

Jul 2, 2008

Can You Remove a Site From a Country Specific Google Search Engine?

Jun 20, 2008

Can Google's First Click Free Program be Used for Web Search?

Jun 5, 2008

Google Now Crawling Content Behind Forms

Apr 14, 2008

Is Google's AdWords Spider Lowercasing Destination URLs?

Apr 2, 2008

Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Volatility, Helpful Content Update Gone, Dangerous Search Results & Ads Confusion - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 15, 2024

Apr 15, 2024 - 4:00 pm
Google Search Engine Optimization

Google Goes On Defensive On Its Search Quality & Forum Results Statements

Apr 15, 2024 - 7:51 am
Google Search Engine Optimization

Google Responds To The Verge Mocking Its Search Rankings For Best Printer

Apr 15, 2024 - 7:41 am
Google News

Google Threatens California: Tests Removing Links To Publishers & Pauses Investments

Apr 15, 2024 - 7:31 am
Google Search Engine Optimization

Google Crawl Budget Is Allocated By Hostname

Apr 15, 2024 - 7:21 am
Google AdSense

Google AdSense Publishers Reporting Huge RPM Earnings Drops

Apr 15, 2024 - 7:11 am