A thread at WebmasterWorld has a post where a member said he spoke with an Ask Jeeves engineer and they told him that they might be adding a parameter to tell the teoma spiders how "hard" to crawl. Now this is supposedly different then the Yahoo! Crawl Delay where, "You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server."

I am trying to find out more information about the validity of this WebmasterWorld thread and what it really means. In the thread, the member said the Ask Jeeves representative said that this is not a Crawl Delay. At this point, I do not see how a hard crawl would be anything else but a time delay.

Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!