Anthropic updates its crawler documentation explaining what its crawlers do, and what blocking them will result in. Anthropic has three primary bots and they include ClaudeBot, Claude-User and Claude-SearchBot.
The documentation is over here and was updated I believe last Friday, February 20th. Pedro Dias spotted the change and posted about it on X saying, "Seems Anthropic today updated their docs to include more information about their crawlers and their purpose."
Here is what it says today:
- ClaudeBot: ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training. When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.
- Claude-User: Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners to control which sites can be accessed through these user-initiated requests. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
- Claude-SearchBot: Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
Anthropic also supports the Crawl-delay directive and robots.txt file directive.
Forum discussion at X.
