Google announced yesterday as part of its efforts to standardizing the robots exclusion protocol that it is open sourcing its robots.txt parser. That means how GoogleBot reads and listens to robots.txt files will be available for any crawler or coder to look at or use.
It is rare for Google to share anything they do in core search with the open source world - it is their secret sauce - but here Google has published it to Github for all to access.
Google wrote they "open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense."
It's been awesome working with @methode and https://t.co/CPJfDQnxn1 on this. I am very happy that it is finally ready to be shared with you all! 😃 https://t.co/gyxvzrFLtp
— Edu Pereda (@epere4) July 1, 2019
If you have SERIOUS ideas about what else could be useful as OSS, leave a comment with the idea and an explanation how would you use that OSS https://t.co/cxxqhI9Nzo
— Gary "鯨理" Illyes (@methode) July 1, 2019
I helped write some of the earliest parts of this code from 1999-2002. Lots of fun times:
— Jeff Dean (@JeffDean) July 2, 2019
What should you do with robots.txt files in MS Word format?
One site had:
User-agent *
Disallow: /
Instead of:
User-agent: *
Disallow: /
(We made our parser less strict) https://t.co/7FnX8lFKqu
Forum discussion at Twitter.
