Google Robots.txt Parser Updated In GitHub

May 23, 2024 - 7:51 am 1 by
Filed Under Google

Google Spider Web

Google has updated its open source robots.txt parser code on GitHub the other day. Gary Illyes from Google pushed the update yesterday morning to the repository there. Google originally released the parser to the world back in 2019.

Gary Illyes explained on LinkedIn that this updated parser has been being used by Google for a while now, but now Google has released that update to GitHub.

Gary wrote, "The release introduces new capabilities in the parser class that allows you to export parsing information about the passed in robotstxt body, and adds a new library to access that information. This new library has been used by Google Search Console for many moons now (in conjunction with the Java port) and so far we haven't encountered issues with it; if you do, file an issue on GitHub!"

When Google initially released this parser, Google wrote they "open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense."

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Web Analytics

Google Analytics Real Time Reporting Broken Again

Jul 17, 2025 - 10:40 am
Search Forum Recap

Daily Search Forum Recap: July 17, 2025

Jul 17, 2025 - 10:00 am
Google

Google AI Mode Gets Gemini 2.5 Pro & Deep Search For A Fee

Jul 17, 2025 - 7:51 am
Google

Google Discover Officially Gets AI-Generated Summaries

Jul 17, 2025 - 7:41 am
Google

How The Google Ask For Me Feature Works & New AI Local Calling

Jul 17, 2025 - 7:31 am
Google

Google Tests Much Larger Favicons In Search

Jul 17, 2025 - 7:21 am
Previous Story: Google Ads Referral / Affiliate Program