Ask Jeeves to Add a Parameter for "Hard Crawl"

Feb 10, 2005 • 10:18 am | comments (1) by twitter Google+ | Filed Under Ask.com - Ask Jeeves
 

A thread at WebmasterWorld has a post where a member said he spoke with an Ask Jeeves engineer and they told him that they might be adding a parameter to tell the teoma spiders how "hard" to crawl. Now this is supposedly different then the Yahoo! Crawl Delay where, "You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server."

I am trying to find out more information about the validity of this WebmasterWorld thread and what it really means. In the thread, the member said the Ask Jeeves representative said that this is not a Crawl Delay. At this point, I do not see how a hard crawl would be anything else but a time delay.

Previous story: Major Yahoo Update Reported
 

Comments:

Barry Schwartz

02/10/2005 08:54 pm

Ask Jeeves Official Response: We support the crawl-delay feature which allows us to fine tune the load imposed by our crawler on web site servers. This is the specification method in robots.txt (for a 10 second delay between requests): Crawl-Delay: 10 Thanks, Kaushal Search Product Manager, Ask Jeeves

blog comments powered by Disqus