Google's John Mueller posted saying, "Authentication for your personal crawlers & SEO tools is going to be more and more of a topic." I believe, but I can be wrong, this is alluding to the num parameter being disabled and how tools are having issues now with scraping Google.
He wrote on Bluesky:
Authentication for your personal crawlers & SEO tools is going to be more and more of a topic. This is a simple guide to get started (though I imagine it'll evolve significantly over time).The neat thing is that you don't have to list your IP addresses, and can be verified from where-ever you are (provided you keep the signature). Ideally you'd authenticate on more than just the signature, so that people can't fake you, but this is early steps.
If you'd like to be involved in the web standards around this, follow datatracker.ietf.org. and the associated mailing list. Your feedback on all of this is welcome there, no need to be an "expert" (whatever that is).
An example of something that's still uncertain: if you have an AI system in your browser that clicks around and does stuff for you, is that a bot that could (should?) be authenticated? or is that just "you"? (a better / different / assisted "you"?)
I mean, is there a way these third party tools can authenticate themselves so they can go beyond 20 results? I suspect not?
Forum discussion at Bluesky.
Note: This was pre-written and scheduled to be posted today, I am currently offline for Rosh Hashanah.
