Google Adds More Details To Robots.txt Specifications Help Document

Apr 28, 2021 • 7:11 am | comments (1) by | Filed Under Google Search Engine Optimization
 

Googlebot

Google has added a bunch of new supporting details to the robots.txt specification document. Jamie Indigo noticed these changes and explained what was added include "explicit details on the files for IDNs, IP-addresses, and port numbered hostnames."

It seems these were added on April 7th according to the document. Jamie posted the details on Twitter saying "Robots.txt docs are full of new delights thanks to @methode and @LizziHarvey's on-going work to make robots protocol an internet standard. The latest additions include explicit details on the files for IDNs, IP-addresses, and port numbered hostnames."

click for full size

Google last did a big update to this document in July 2019.

Forum discussion at Twitter.

Previous story: Daily Search Forum Recap: April 27, 2021
Ninja Banner
 
blog comments powered by Disqus