Ask Jeeves to Add a Parameter for "Hard Crawl"

Feb 10, 2005 - 10:18 am 1 by
Filed Under Ask.com

A thread at WebmasterWorld has a post where a member said he spoke with an Ask Jeeves engineer and they told him that they might be adding a parameter to tell the teoma spiders how "hard" to crawl. Now this is supposedly different then the Yahoo! Crawl Delay where, "You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server."

I am trying to find out more information about the validity of this WebmasterWorld thread and what it really means. In the thread, the member said the Ask Jeeves representative said that this is not a Crawl Delay. At this point, I do not see how a hard crawl would be anything else but a time delay.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 17, 2024

May 17, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More

May 17, 2024 - 8:01 am
Google Search Engine Optimization

Remove Your Content From Google's AI Overviews

May 17, 2024 - 7:51 am
Google Ads

Google Ads AI Summaries Live For Some Advertisers

May 17, 2024 - 7:41 am
Google Maps

Order with Google For Food Delivery Going Away End Of June

May 17, 2024 - 7:31 am
Google Search Engine Optimization

Two New Googlebots: GoogleOther-Image & GoogleOther-Video

May 17, 2024 - 7:21 am
Previous Story: Major Yahoo Update Reported