OpenAI's ChatGPT New Web Crawler - GPTBot

Aug 7, 2023 - 7:21 am 1 — by
Filed Under ChatGPT

Robot Block

OpenAI, the folks behind ChatGPT, have published information on its web crawler named GPTBot. You can now see if OpenAI is crawling your site, how much so, and you can disallow access to all or part of your site with the robots.txt protocol.

You can see the documentation for GPTBot over here.

  • User agent token: GPTBot
  • Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

You can then disallow using the user-agent GPTBot like you would any other crawler.

Currently, the IP range listed for GPTbot is just 40.83.2.64/28 but that can change, so check that file for updates.

OpenAI lists GPTBot's usage as, "Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies. Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety. Below, we also share how to disallow GPTBot from accessing your site."

Yesterday, I spotted a new thread at WebmasterWorld with complaints about GPTBot activity. The webmaster said, "Just had over 1000 hits from this bot, hitting individual pages. As it happens my site automatically served a 403 for each hit because the bot is not in my whitelist, nor did it pass the 'human' test."

Previously, you were only able to block ChatGPT plugins. And it seems like Google and others are working on an alternative to robots.txt for AI search purposes.

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: November 20, 2026

Nov 20, 2026 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google November Core Update Status, FBI Alert On SEO Poisoning, Bing & ChatGPT Web Only Links, Microsoft Ads Bid Upgrade

Nov 20, 2026 - 8:01 am
Google

Google, ChatGPT & Others AI Engines Provide Coupon Codes

Nov 20, 2026 - 7:51 am
ChatGPT

ChatGPT Card Design

Nov 20, 2026 - 7:41 am
Google Ads

Google Expands & Clarifies Sexual Content Advertising Policy Examples

Nov 20, 2026 - 7:31 am
Bing Ads

Bing & Microsoft Ads - Made With AI Label

Nov 20, 2026 - 7:21 am
Previous Story: Google Ads Releases New Responsive Search Ads Guide (PDF)