Entries from Search Engine Roundtable tagged with 'googlebot'

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said: After...

Posted on November 20, 2009 8:52 AM

A Google Webmaster Help thread has reports that the fetch as Googlebot feature might display weird characters for some pages that return non-ASCII characters. Specifically, if the pages are not encoded in UTF-8 and use these non-ASCII characters, the tool...

Posted on October 21, 2009 8:36 AM

Google added the fetch as Googlebot feature the other day and now people are really beginning to explore it. One topic I have seen come up was why is the Fetch as Googlebot feature only showing up to 100Kb of...

Posted on October 14, 2009 8:33 AM

The Google Webmaster Central Blog announced the launch of a new "Labs" section in Google Webmaster Tools. Labs is for Google to launch features that might not be fully tested and have bugs, but at the same time give webmasters...

Posted on October 13, 2009 8:45 AM

A Google Webmaster Help thread has an interesting discussion around blocking your site from coming up for both visitors and search engine crawlers on Shabbat (the Jewish Saturday). This is not a new topic, we discussed using cloaking for religious...

Posted on September 9, 2009 8:57 AM

JohnMu from Google posted in a Google Webmaster Help thread that Google typically crawls a site's robots.txt file on a daily basis. This is the first time (at least that I can remember) I have seen a Googler make a...

Posted on September 3, 2009 8:42 AM

Back in the day, tracking how bots accessed your site was a bit of a crave. Now, you don't hear about it much. The old Google Analytics, aka Urchin, had a section for displaying bot activity on your site. It...

Posted on April 6, 2009 10:08 AM

I found a fun but serious SEO mistake in a Google Webmaster Help thread that I wanted to share with you all. But I felt it would be fun, if you had the time, to share at least one of...

Posted on December 18, 2008 8:17 AM

A Google Webmaster Help thread has a member upset that Google is crawling his site during times when his server is overloaded. Is there a way to tell GoogleBot to stay away during these times? JohnMu of Google said, yes...

Posted on December 12, 2008 8:25 AM

It seems like we have confirmed reports from a Googler in Google Groups that Google's video crawler, part of the GoogleBot family, is not playing nice. In short, even though you may be telling Google not to crawl your videos,...

Posted on December 1, 2008 8:14 AM

I found an interesting tidbit while reading a somewhat detailed thread at Google Groups. The scenario is as follows. You have blocked Googlebot from accessing your site for a 6 month period or so. Then you want to welcome Googlebot...

Posted on November 14, 2008 7:07 AM

A webmaster at Google Groups may have uncovered an issue with a specific firewall program named CSF (ConfigServer Security and Firewall). In short, this webmaster found that his host used this firewall and it was blocking GoogleBot from accessing his...

Posted on November 3, 2008 7:39 AM

A Google Groups thread has a webmaster questioning the date shown in Google Webmaster Tools that shows the "Home page crawl" summary. In short, the data may be older and slower to update than other tools you have in your...

Posted on September 24, 2008 8:57 AM

A Google Groups thread has a detailed discussion around the topic of Google spider, GoogleBot, crawling too much. Sometimes servers can be overwhelmed by all the traffic it gets and automated crawlers, such as GoogleBot, can add a tremendous amount...

Posted on August 28, 2008 8:01 AM

There are threads at Google Groups and DigitalPoint Forums with multiple reports of Google not crawling Blogger hosted blogs, that are on custom or private domains (i.e. not on blogspot.com domains). Many have reported that the Googlebot crawling has stopped...

Posted on August 7, 2008 8:26 AM

Last night, I had a nice chat with Googler, JohnMu. I joked around with John, asking if he has messed up yet, in terms of Google communication with webmasters. He said not really - which I agree with. But he...

Posted on June 5, 2008 10:05 AM

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM

We should have seen this coming, based on the number of reports that Google was submitting GET forms. But often, it is hard to validate those types of reports, due to people spoofing Googlebot and similar tactics. In any event,...

Posted on April 14, 2008 7:50 AM

Brendan Kowitz wrote a blog post about an interesting anomaly he noticed as Googlebot was crawling his site. Apparently, in March of 2006, Googlebot's User-Agent string was changed which triggered a bug in ASP.NET webservers. What this would mean is...

Posted on November 8, 2007 10:14 AM

Susan Moskwa, a member of the Google Webmaster Central team, has said that afraid.org, a free DNS provider, has been known to block GoogleBot. Just found out that your DNS provider (afraid.org) has been known to block Googlebot from certain...

Posted on October 16, 2007 11:17 AM

A WebmasterWorld thread discusses a more detailed issue with how Google's spider, GoogleBot, is crawling some pages. Let me quote the detailed explanation: I've tried: Checking for the HTTP_IF_MODIFIED_SINCE header and returns "304 Not Modified" if possible. Problem: Googlebot doesn't...

Posted on October 9, 2007 7:46 AM

The other day we reported of a Possible GoogleBot DNS Issues Causing Indexing Issues at Google.com. It has now been confirmed as an issue, and is now reportedly fixed. Susan Moskwa of Google said in a Google Groups post: It...

Posted on September 19, 2007 8:03 AM

A detailed Google Groups thread is reporting various reports of webmasters claiming GoogleBot is timing out before reaching their pages. First, these webmasters are noticing a drop in GoogleBot activity on their server. So they login to Google Webmaster Tools...

Posted on September 11, 2007 8:12 AM

Ever wanted to become a Googlebot to see what the bot sees? Now you can. While a Firefox Extension already does this, a DigitalPoint Forums thread discusses how to do this as well (for PC users). The idea is to...

Posted on July 23, 2007 9:37 AM

The Google Webmaster Central blog announced the release of several new tools added to the Webmaster Central toolbox. The new features are pretty neat and include; Googlebot activity reports that shows the "number of pages Googlebot's crawled from your site...

Posted on October 19, 2006 7:24 AM