OpenAI's ChatGPT New Web Crawler - GPTBot

Aug 7, 2023 - 7:21 am 1 by

Robot Block

OpenAI, the folks behind ChatGPT, have published information on its web crawler named GPTBot. You can now see if OpenAI is crawling your site, how much so, and you can disallow access to all or part of your site with the robots.txt protocol.

You can see the documentation for GPTBot over here.

  • User agent token: GPTBot
  • Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

You can then disallow using the user-agent GPTBot like you would any other crawler.

Currently, the IP range listed for GPTbot is just 40.83.2.64/28 but that can change, so check that file for updates.

OpenAI lists GPTBot's usage as, "Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies. Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety. Below, we also share how to disallow GPTBot from accessing your site."

Yesterday, I spotted a new thread at WebmasterWorld with complaints about GPTBot activity. The webmaster said, "Just had over 1000 hits from this bot, hitting individual pages. As it happens my site automatically served a 403 for each hit because the bot is not in my whitelist, nor did it pass the 'human' test."

Previously, you were only able to block ChatGPT plugins. And it seems like Google and others are working on an alternative to robots.txt for AI search purposes.

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: October 3, 2024

Oct 3, 2024 - 10:00 am
Google Updates

Record-Breaking Google Search Ranking Volatility Continues Into October

Oct 3, 2024 - 8:01 am
Bing Ads

Microsoft Advertising New Ad Experience, Diagnostics & Performance Snapshot

Oct 3, 2024 - 7:51 am
Google

Google Testing Search Instead For Section

Oct 3, 2024 - 7:41 am
Google

Google: SEOs Can Help Shape Client Policies & Decisions On AI Bots

Oct 3, 2024 - 7:31 am
Google Ads

Google Ads With Images & Map Carousel

Oct 3, 2024 - 7:21 am
Previous Story: Google Ads Releases New Responsive Search Ads Guide (PDF)