Entries from Search Engine Roundtable tagged with 'googlebot'

Last night, I had a nice chat with Googler, JohnMu. I joked around with John, asking if he has messed up yet, in terms of Google communication with webmasters. He said not really - which I agree with. But he...

Posted on June 5, 2008 10:05 AM

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM

We should have seen this coming, based on the number of reports that Google was submitting GET forms. But often, it is hard to validate those types of reports, due to people spoofing Googlebot and similar tactics. In any event,...

Posted on April 14, 2008 7:50 AM

Brendan Kowitz wrote a blog post about an interesting anomaly he noticed as Googlebot was crawling his site. Apparently, in March of 2006, Googlebot's User-Agent string was changed which triggered a bug in ASP.NET webservers. What this would mean is...

Posted on November 8, 2007 10:14 AM

Susan Moskwa, a member of the Google Webmaster Central team, has said that afraid.org, a free DNS provider, has been known to block GoogleBot. Just found out that your DNS provider (afraid.org) has been known to block Googlebot from certain...

Posted on October 16, 2007 11:17 AM

A WebmasterWorld thread discusses a more detailed issue with how Google's spider, GoogleBot, is crawling some pages. Let me quote the detailed explanation: I've tried: Checking for the HTTP_IF_MODIFIED_SINCE header and returns "304 Not Modified" if possible. Problem: Googlebot doesn't...

Posted on October 9, 2007 7:46 AM

The other day we reported of a Possible GoogleBot DNS Issues Causing Indexing Issues at Google.com. It has now been confirmed as an issue, and is now reportedly fixed. Susan Moskwa of Google said in a Google Groups post: It...

Posted on September 19, 2007 8:03 AM

A detailed Google Groups thread is reporting various reports of webmasters claiming GoogleBot is timing out before reaching their pages. First, these webmasters are noticing a drop in GoogleBot activity on their server. So they login to Google Webmaster Tools...

Posted on September 11, 2007 8:12 AM

Ever wanted to become a Googlebot to see what the bot sees? Now you can. While a Firefox Extension already does this, a DigitalPoint Forums thread discusses how to do this as well (for PC users). The idea is to...

Posted on July 23, 2007 9:37 AM

The Google Webmaster Central blog announced the release of several new tools added to the Webmaster Central toolbox. The new features are pretty neat and include; Googlebot activity reports that shows the "number of pages Googlebot's crawled from your site...

Posted on October 19, 2006 7:24 AM