Google Search Crawl Rate Doc Adds Retry-After HTTP & More

Oct 6, 2026 - 2:59 pm 0 — by

Google Crawler

Google has just pushed some updates to the reduce the Google crawl rate documentation. Most of the changes are just organizational to the content on the page but under the "Urgently reduce crawler traffic (for emergencies)" section, Google added new information on how to use Retry-After HTTP header.

Google said it has "Restructured the emergency crawl rate reduction section and added information and examples about the Retry-After HTTP header to the Reduce the Google crawl rate documentation." Google added it did this to "Support for the Retry-After HTTP header is not new (it was already documented in Temporarily pause or disable a website), and adding it directly to the crawl rate reduction guide makes it easier to find and understand when urgently reducing crawler traffic."

If you compare the live doc to the archived doc, you will mostly see a lot of things moved around the page.

The biggest addition is the section on:

When returning a 503 or 429 status code, you can also include a Retry-After HTTP header (as defined in RFC 9110 HTTP Semantics) to indicate when Google's crawlers can retry the request, using either a delay in seconds or an absolute UTC date and time:

Google added example code as well, here is a screenshot:

Google Crawl Docs Page

Google isn't introducing a new way to throttle Googlebot, the underlying mechanism hasn't changed. Google already used 500, 503, and 429 responses as signals to reduce crawling. The new documentation makes Retry-After an explicit additional signal that site owners can provide when using 503 or 429.

Forum discussion at X.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Video Recaps

Search News Buzz Video Recap: Google November Core Update Status, FBI Alert On SEO Poisoning, Bing & ChatGPT Web Only Links, Microsoft Ads Bid Upgrade

Nov 20, 2026 - 8:01 am
Google

Google, ChatGPT & Others AI Engines Provide Coupon Codes

Nov 20, 2026 - 7:51 am
ChatGPT

ChatGPT Card Design

Nov 20, 2026 - 7:41 am
Google Ads

Google Expands & Clarifies Sexual Content Advertising Policy Examples

Nov 20, 2026 - 7:31 am
Bing Ads

Bing & Microsoft Ads - Made With AI Label

Nov 20, 2026 - 7:21 am
Search News

The .seo Top-Level Domain (TLD) May Be Coming Soon

Nov 20, 2026 - 7:11 am
Previous Story: Nano Banana 2.1 In Google AI Mode In Search