Entries from Search Engine Roundtable tagged with 'crawling'

Google Crawl Rate Drops: Google Responds?

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints. This time, I decided to...

GoogleBot Getting Tired? Google's Spider Crawling Less Documents

A WebmasterWorld thread reports from dozens of Webmasters that GoogleBot, Google's web crawler has not been crawling as many documents as they have in the past. Many webmasters are noticing reduction in crawl rates as much as 90-percent, relative to...

Managing Duplicate Content In a World Where Google Can Crawl JavaScript

Now that Google admitted to crawling JavaScript and forms SEOs and Webmasters need to be aware of how to manage even more duplicate content issues. In the past, a good strategy was to build out filter pages (filter by color,...

Wordpress Installation Now Blocking Search Engines?

A WebmasterWorld thread reports that new installations of the popular blogging software, WordPress, is by default blocking all search engines. He said, when you go to the Privacy Options section in the administration panel, by default, it is set to...

Google's Set Crawl Rate Feature Works at Domain or Sub Domain Only

A Google Groups thread has a fairly simple but educational FAQ on how the "Set Crawl Rate" feature works in Google Webmaster Tools. In short, you can only set the crawl rate for a site on the domain or subdomain...

Yahoo Slurp Taking a Break? Reported Slow Crawling Activity

In August, Yahoo announced a new crawl behavior for Slurp, Yahoo's web crawler. The new crawl behavior was suppose to tame the crawler to go through your site in a more relaxed and efficient manner for both the crawler and...

How To Ask GoogleBot (Google) To Crawl Your Site

Last week Tamar wrote about How to Stop Googlebot from Crawling Your Site Rapidly, so I thought I write about the opposite. How can you induce GoogleBot into crawling your site. Although there is no magic shot that guarantees inducement...

GoogleBot Not Sending IF_MODIFIED_SINCE Request?

A WebmasterWorld thread discusses a more detailed issue with how Google's spider, GoogleBot, is crawling some pages. Let me quote the detailed explanation: I've tried: Checking for the HTTP_IF_MODIFIED_SINCE header and returns "304 Not Modified" if possible. Problem: Googlebot doesn't...

Possible GoogleBot DNS Issues Causing Indexing Issues at Google.com

A detailed Google Groups thread is reporting various reports of webmasters claiming GoogleBot is timing out before reaching their pages. First, these webmasters are noticing a drop in GoogleBot activity on their server. So they login to Google Webmaster Tools...

Managing the Robots.txt File for Sites Sharing Same Local Files

A Cre8asite Forums thread asks how can he generate unique robots.txt files for each domain he has, when each of those sites are sharing the same local files through a form of IIS mirroring? There are several ways to do...

Made for AdSense Sites Can Get You Delisted

A very interesting Google Groups thread has many bloggers, including SEO Buzz Box, DaveN, and Search Engine Journal voicing their reactions. The background is that the webmaster of AlkenMRS.com realized that his 10+ year old site had been delisted from...


To subscribe to the Search Engine Roundtable, click here