Google Robots.txt Parser Updated In GitHub

May 23, 2024 - 7:51 am 1 by
Filed Under Google

Google Spider Web

Google has updated its open source robots.txt parser code on GitHub the other day. Gary Illyes from Google pushed the update yesterday morning to the repository there. Google originally released the parser to the world back in 2019.

Gary Illyes explained on LinkedIn that this updated parser has been being used by Google for a while now, but now Google has released that update to GitHub.

Gary wrote, "The release introduces new capabilities in the parser class that allows you to export parsing information about the passed in robotstxt body, and adds a new library to access that information. This new library has been used by Google Search Console for many moons now (in conjunction with the Java port) and so far we haven't encountered issues with it; if you do, file an issue on GitHub!"

When Google initially released this parser, Google wrote they "open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense."

Forum discussion at LinkedIn.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Weekend Volatility, Google On Search Leak, Elizabeth Tucker Interview & Apple Intelligence - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles


Google Search Bug: Not Indexing or Serving New Content?

Jun 20, 2024 - 6:56 pm
Search Forum Recap

Daily Search Forum Recap: June 20, 2024

Jun 20, 2024 - 10:00 am
Google Search Engine Optimization

Google Warns On Using JavaScript For Structured Data

Jun 20, 2024 - 7:51 am

Most Google People Also Ask Responses Are Wikipedia

Jun 20, 2024 - 7:41 am
Google Maps

Google Map Pin Exploit Leads To Local Rankings Drop & Possible Suspension

Jun 20, 2024 - 7:31 am

Google Search Tests Latest Deals With Deals In Past 24 Hours

Jun 20, 2024 - 7:21 am
Previous Story: Google Ads Referral / Affiliate Program