OpenAI Releases ChatGPT Search: Citations, Robots.txt Controls & More

Nov 1, 2024 - 7:41 am 4 by

Openai Chatgpt Search Web

After beta testing SearchGPT for a few months, OpenAI is now rolling out Search within ChatGPT for ChatGPT Plus and Team users, which will come to free users in the future (if you signed up for the waitlist for SearchGPT, you should also get it for free).

The first thing most of us will look at are the citations, links, will Search in ChatGPT drive searchers to our sites. OpenAI wrote, "Chats now include links to sources, such as news articles and blog posts, giving you a way to learn more. Click the Sources button below the response to open a sidebar with the references."

Here is what that looks like:

Chatgpt Search Citations

I did some searches for content that this site might rank for and the results were not half bad, although not perfect but what search engine is?

Chatgpt Search Example

The citation button brings up these, respectively:

Chatgpt Search Citations1

Chatgpt Search Citations2

Sports query:

Chatgpt Search Example2

You can also opt out if you don't want in Search within ChatGPT. OpenAI published its bot controls which lists three bots:

  • OAI-SearchBot: OAI-SearchBot is for search: Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
  • ChatGPT-User: ChatGPT-User is for user actions in ChatGPT and Custom GPTs: Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
  • GPTBot: GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models : Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

"OpenAI uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request. OpenAI uses the following robots.txt tags to enable webmasters to manage how their sites and content work with AI. Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTbot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models. For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust," OpenAI wrote.

There are screenshots on OpenAI's blog post describing why ChatGPT does search better, so make sure to review that. OpenAI says ChatGPT Search draws on articles from the WSJ and others it has deals with, and will direct election-related queries to sources like AP and Reuters. Some of the other partners listed include, Associated Press, Axel Springer, Condé Nast, Dotdash Meredith, Financial Times, GEDI, Hearst, Le Monde, News Corp, Prisa (El País), Reuters, The Atlantic, Time, and Vox Media.

Time will tell if this makes Google dance and has any impact on Google's market share. So far, Microsoft didn't really make any dents with search market share when using OpenAI's technology in their search engine.

Also, the results probably won't be consistent at all across next search or other searchers because of how these language models work:

And smart move OpenAI:

Sam Altman did an AMA on Reddit about Search in ChatGPT.

Forum discussion at Reddit and WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: December 2, 2024

Dec 2, 2024 - 10:00 am
Blog Administration

21 Years Covering The Search Industry

Dec 2, 2024 - 8:01 am
Google Updates

December 2024 Google Webmaster Report

Dec 2, 2024 - 7:51 am
Google

Google Chrome Site Engagement Metrics

Dec 2, 2024 - 7:41 am
Google

Google Math Solver Watch Animated Tutorials

Dec 2, 2024 - 7:31 am
Bing Search

Bing Tests Shopping / E-Commerce Card

Dec 2, 2024 - 7:21 am
Previous Story: Google AI Answers In Google Maps: Things To Do, Review Summaries & About This Place