OpenAI made a number of clarification changes to its crawler help documentation to say that version number of the user-agent string may be updated and it may use a user-agent string with an additional robots.txt marker.
The example given of the user-agent string in the doc, OpenAI added "the version number may change."
Plus, OpenAI added a section to say:
When fetching robots.txt files, we may use a user-agent string with an additional robots.txt marker. The marker helps site owners more easily distinguish requests for the robots.txt file from requests for other resources, especially when their logs do not include paths: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
I don't think any of this is really novel but it was updated in the help documentation, so I wanted to make sure you all saw this.
Here are the changes documented from the previous version I checked:
Forum discussion at X.
