OpenAI has updated its spider/crawler it uses for ChatGPT, OAI-SearchBot. The AI company updated the description and details of the crawler information removing that it is used for links and to train OpenAI's generative AI foundation models.
Pieter Serraris spotted this change and posted about it on LinkedIn - he wrote, "OpenAI has made some big changes in the description of the roles of their crawlers and user agents."
Here are the changes, the words highlighted in red were removed and the words highlighted in green were added:
Pieter summarized the changes:
- It looks like ChatGPT User (the user action bot) will no longer comply to robots.txt rules (Open changed the wording from "the following robots.txt tags", referring to all 3 user agents to "OAI SearchBot and GPTBot robots.txt tags")
- OAISearchBot is no longer used to feed the navigational links in ChatGPT answers (blocking this bot does not mean your will not appear in the links)
- ChatGPT User is now also said to be used for Custom GPT requests and GPT Actions.
- OAISearchBot and GPTBot share information with each other ("If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicate crawling.") -> this confirms what we have been seeing lately in the log files: for some prompts we see 2 or all 3 OAI bots visiting the site.
For most people, OpenAI crawls your site way more than it sends you any traffic - so keep that in mind.
Forum discussion at LinkedIn.
