Google Adds More Details To Robots.txt Specifications Help Document

Apr 28, 2021 - 7:11 am 1 by

Googlebot

Google has added a bunch of new supporting details to the robots.txt specification document. Jamie Indigo noticed these changes and explained what was added include "explicit details on the files for IDNs, IP-addresses, and port numbered hostnames."

It seems these were added on April 7th according to the document. Jamie posted the details on Twitter saying "Robots.txt docs are full of new delights thanks to @methode and @LizziHarvey's on-going work to make robots protocol an internet standard. The latest additions include explicit details on the files for IDNs, IP-addresses, and port numbered hostnames."

click for full size

Google last did a big update to this document in July 2019.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 17, 2024

May 17, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More

May 17, 2024 - 8:01 am
Google Search Engine Optimization

Remove Your Content From Google's AI Overviews

May 17, 2024 - 7:51 am
Google Ads

Google Ads AI Summaries Live For Some Advertisers

May 17, 2024 - 7:41 am
Google Maps

Order with Google For Food Delivery Going Away End Of June

May 17, 2024 - 7:31 am
Google Search Engine Optimization

Two New Googlebots: GoogleOther-Image & GoogleOther-Video

May 17, 2024 - 7:21 am
Previous Story: Google Logo Fallen & In Pieces