Spiders

Older Entries

GoogleBot Getting Tired? Google's Spider Crawling Less Documents

May 1, 2008

Google Now Crawling Content Behind Forms

Apr 14, 2008

Is Google's AdWords Spider Lowercasing Destination URLs?

Apr 2, 2008

Yahoo Slurp, Yahoo Search Crawler, Suffering From ADHD?

Mar 31, 2008

MSNBot Again Failing Reverse DNS Test

Mar 11, 2008

MSRBot Bot, Microsoft Bot Causing Havoc on Some Servers?

Feb 25, 2008

Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler

Feb 13, 2008

Can You Hide Text & Links From Your Users, If Search Engines Won't See?

Jan 22, 2008

Google's Remove URLs Feature Removes For 90 Days Only

Jan 22, 2008

Wordpress Installation Now Blocking Search Engines?

Jan 22, 2008

Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP?

Jan 14, 2008

Ask.com Fixes Crawler Issue With Badly-Formed URLs

Jan 4, 2008

Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors?

Dec 31, 2007

Microsoft Confirms Reverse DNS Bot Issue & Promises To Resolve

Dec 6, 2007

MSNBot Reverse DNS Test Fails Requirements

Dec 5, 2007

Live Search's Spam Fake Referrals Were Cloaking Tests, Says Microsoft

Dec 4, 2007

Double Check Your Robots.txt: Google Testing New Crawler Directives

Nov 21, 2007

Got ASP 2.0? GoogleBot Can Trip a Bug in the Code

Nov 8, 2007

Google Says DNS Provider Afraid.org Blocks GoogleBot

Oct 16, 2007

GoogleBot Not Sending IF_MODIFIED_SINCE Request?

Oct 9, 2007

Googlebot DNS Caching Issue Reportedly Resolved

Sep 19, 2007

Possible GoogleBot DNS Issues Causing Indexing Issues at Google.com

Sep 11, 2007

Is a Robots.txt File Required for Search Engine Optimization?

Aug 31, 2007

Managing the Robots.txt File for Sites Sharing Same Local Files

Aug 30, 2007

Becoming a Googlebot for Testing Purposes

Jul 23, 2007

Does Googlebot Index Faster When Integrating Google Custom Search?

Jul 16, 2007

Yahoo! Supports Robots-Nocontent: Enabling Organic Search Page Section Targeting

May 3, 2007

Verify The Bots Accessing Your Site: Is Google.com Sending That GoogleBot?

Mar 7, 2007

Google & Search Engines Do Not Mind Bad HTML When Crawling

Dec 25, 2006

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: March 15, 2024

Mar 15, 2024 - 4:00 pm