Entries from Search Engine Roundtable tagged with 'crawling'

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

Posted on May 2, 2008 8:00 AM

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Posted on May 1, 2008 7:42 AM

Now that Google admitted to crawling JavaScript and forms SEOs and Webmasters need to be aware of how to manage even more duplicate content issues. In the past, a good strategy was to build out filter pages (filter by color,...

Posted on April 30, 2008 8:01 AM

A WebmasterWorld thread reports that new installations of the popular blogging software, WordPress, is by default blocking all search engines. He said, when you go to the Privacy Options section in the administration panel, by default, it is set to...

Posted on January 22, 2008 6:53 AM

A Google Groups thread has a fairly simple but educational FAQ on how the "Set Crawl Rate" feature works in Google Webmaster Tools. In short, you can only set the crawl rate for a site on the domain or subdomain...

Posted on October 25, 2007 8:15 AM

In August, Yahoo announced a new crawl behavior for Slurp, Yahoo's web crawler. The new crawl behavior was suppose to tame the crawler to go through your site in a more relaxed and efficient manner for both the crawler and...

Posted on October 17, 2007 6:38 AM

Last week Tamar wrote about How to Stop Googlebot from Crawling Your Site Rapidly, so I thought I write about the opposite. How can you induce GoogleBot into crawling your site. Although there is no magic shot that guarantees inducement...

Posted on October 11, 2007 7:35 AM

A WebmasterWorld thread discusses a more detailed issue with how Google's spider, GoogleBot, is crawling some pages. Let me quote the detailed explanation: I've tried: Checking for the HTTP_IF_MODIFIED_SINCE header and returns "304 Not Modified" if possible. Problem: Googlebot doesn't...

Posted on October 9, 2007 7:46 AM

A detailed Google Groups thread is reporting various reports of webmasters claiming GoogleBot is timing out before reaching their pages. First, these webmasters are noticing a drop in GoogleBot activity on their server. So they login to Google Webmaster Tools...

Posted on September 11, 2007 8:12 AM

A Cre8asite Forums thread asks how can he generate unique robots.txt files for each domain he has, when each of those sites are sharing the same local files through a form of IIS mirroring? There are several ways to do...

Posted on August 30, 2007 7:03 AM

A very interesting Google Groups thread has many bloggers, including SEO Buzz Box, DaveN, and Search Engine Journal voicing their reactions. The background is that the webmaster of AlkenMRS.com realized that his 10+ year old site had been delisted from...

Posted on July 11, 2007 10:01 AM