Apple has made some updates to its Applebot documentation, primarily explaining the difference between the normal Applebot crawler and the Applebot-Extended crawler. To be clear, Applebot-Extended is not new, Apple introduced it about a year ago but Apple wanted to clarify how blocking Applebot-Extended impacts Applebot, if at all.
Apple explained that "data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools."
Now, if you want to block Applebot-Extended, Apple wrote, "Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file."
But even if you do block it, Applebot may still be used for other Apple Search related services. Apple wrote, "Even if you disallow Applebot-Extended, your website instructions may still allow Applebot to crawl your webpages. In that case, your content would remain discoverable through Spotlight, Siri, as well as other system-wide features on Apple devices."
There was another small change but not sustainable, you can view the older version here and the new version here.
Here is the big change in the new version:
Here is what it read before:
Forum discussion at X.
