Google Shares Its Robots.txt Parser Code With Open Source World

Jul 2, 2019 • 7:46 am | comments (0) by twitter | Filed Under Google Search Engine
 

Google announced yesterday as part of its efforts to standardizing the robots exclusion protocol that it is open sourcing its robots.txt parser. That means how GoogleBot reads and listens to robots.txt files will be available for any crawler or coder to look at or use.

It is rare for Google to share anything they do in core search with the open source world - it is their secret sauce - but here Google has published it to Github for all to access.

Google wrote they "open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense."

Forum discussion at Twitter.

Previous story: Search Google For Fireworks & Get A Fireworks Show
 
blog comments powered by Disqus