Apple has made changes to its documentation for AppleBot to include crawling and usage for its AI efforts. There were other changes made to the document on June 08, 2026 as well. Those include the additions of nosnippet controls, crawl delay, X-Robots-Tag, accessibility and more.
The About Applebot document added a new section on AI, it reads:
Applebot crawled data may be used to provide additional context and up-to-date content when AI models are used to generate output for display in Apple products and services. For example, answering broad world knowledge questions in Siri and Search that may include links to sources and websites used to help generate the answer. Web publishers can opt out of their content being used in these broad world knowledge answers by applying the nosnippet meta tag to specific content.
In addition to the AI updates, Apple also added details about nosnippet controls. Here are the sections that were updated related to that:
Even if you disallow Applebot-Extended and tag website content with the nosnippet meta tag, your website instructions may still allow Applebot to crawl your webpages. Your content will remain discoverable through Spotlight, Siri, and Safari, as well as other system-wide features on Apple devices.
Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services.
There is also a section on crawl delay that was added, specifically that Applebot does not follow crawl-delay.
Applebot does not follow crawl-delay.Applebot is engineered for efficiency and will adjust to minimize the impact on site owners. For example, to avoid overloading site servers, Applebot’s crawl rate adjusts automatically when a site slows down or returns errors. Apple also caches crawled content to reduce unnecessary crawling. Identifying content that doesn't need to be crawled lowers infrastructure costs for site owners and makes the internet more efficient as a whole.
There is a new section on X-Robots-Tag:
Using the X-Robots-Tag HTTP header Applebot also supports indexing directives delivered via the X-Robots-Tag HTTP response header. This is useful for non-HTML resources (such as PDFs or images) where meta tags can’t be used, or when you want to apply directives at the server level.To apply a directive to Applebot: X-Robots-Tag: applebot: nosnippet Marking paywalled content Applebot supports the schema.org isAccessibleForFree property to identify pages that are behind a paywall, metered access, or subscription. Add the property to your page-level structured data in JSON-LD:
And a new section on Accessibility:
Pages marked isAccessibleForFree: false are eligible to appear in search results, but Applebot will not use that content as additional context when AI models are used to generate output for display in Apple products and services. This signal applies at the page level. Section-level markup using hasPart is not supported. To opt out of having your content used to train Apple’s foundation models, use Applebot-Extended described in the next section.
Those are most of the changes to this document.
Forum discussion at X.
