Gary Illyes from Google was recently interviewed by Aneta Samkoff, host of the Odys Podcast. The topic was all about bots and crawlers and they covered a lot of ground. Gary said bot impersonation is more of a problem now because of AI. Bot impersonation is when rogue bots pretend to be someone they are not, like pretending to be Googlebot.
Gary said, "I think it is AI," for the reason why bot impersonation has become such a big issue these days. He said, "so basically more and more companies are looking for tokens for training their own models, for example, and that needs content. And the idea is that basically the internet is a big farm where you can just go out and harvest whatever you need. And firewalls and CDNs in general got better and better identifying these crawlers but they are not perfect. So they still let through some traffic that they meant to stop and just knowing who is crawling you basically enables you to make the decision for yourself about who should access your site and who should not.
The reason I think folks do bot impersonation is so that if you run a smaller AI engine or you want to scrape a site, you can say you are Googlebot and they will let you in. Otherwise, you might be blocked. Of course, there are more ways to validate if the useragent is Googlebot, that it is really Googlebot, such as by reverse DNS verification and IP addresses.
Here is the video:
Here are some of the key points in this video:
- Anyone can impersonate Googlebot in about 30 seconds: Changing a crawler's user-agent header is extremely easy. Illyes says he has set up online honeypots that attract crawlers and found requests claiming to be Googlebot coming from IP addresses unrelated to Google. Website owners should not trust a Googlebot user-agent alone. They should verify the IP address and reverse DNS information.
- AI has made bot impersonation a bigger problem: Gary Illyes attributes the growing concern to companies harvesting internet content to train AI models. Although firewalls and CDNs have improved at detecting fake crawlers, they do not catch everything. Identifying who is crawling a website helps publishers decide which traffic to allow.
- Fake crawlers can overwhelm websites and take them offline: Google has spent years tuning its crawling infrastructure to avoid exhausting website resources. Other crawlers may not have the same safeguards. An improperly configured bot can repeatedly request URLs, consume bandwidth and server resources, and prevent legitimate visitors from accessing a site. Illyes says much of today's damage may result from ignorance or poor implementation rather than deliberate attacks.
- robots.txt is not a security mechanism: Gary Illyes describes robots.txt as a suggestion rather than an enforceable restriction. Legitimate crawlers generally respect it, but bots can ignore it. Website owners who need to block unwanted traffic should use network-level protections, such as a web application firewall (WAF).
- Google already separates search crawling from certain AI uses: Website owners can allow Googlebot to crawl pages for Search while using the Google-Extended robots.txt token to restrict eligible uses of content for training or improving certain Google AI models and services. Illyes notes that Apple has a similar Applebot-Extended token. He expects access controls for different types of automated clients could become more granular.
- Cryptographic signatures could make bot verification more reliable: Gary Illyes discusses Web Bot Auth, an effort involving cryptographically signed HTTP requests. Rather than relying solely on IP addresses or user-agent strings, a website could verify a signature against a public key to check whether a request genuinely comes from the claimed organization. This could simplify authentication as IP addresses become more difficult and expensive to manage.
- Bot reputation scores are a possible future development: Gary Illyes says he is exploring the idea (he works with the robots.txt folks) of a reputation system for crawlers, although it remains an early-stage idea. Such a system might aggregate feedback from websites and security providers about whether bots behave responsibly. He cautions that a system controlled by one company could introduce bias.
- Most small websites do not need to worry about bots immediately: His advice is to determine whether bot traffic is actually causing a problem before investing in protections. For sites that need help, a CDN with bot-management capabilities can be a practical starting point. Hosted platforms such as Shopify already provide protections. Larger or more popular sites may eventually need more sophisticated controls.
Here are some steps you can take to protect yourself from rogue bots:
- Verify Googlebot properly: Check Google's published crawler IP ranges and reverse DNS, rather than trusting the user-agent string alone.
- Monitor server logs: Look for suspicious crawling patterns, unexpected IP addresses and unusually frequent requests.
- Use a CDN or WAF: Apply network-level protections when unwanted bots threaten site performance or security.
- Configure robots.txt: Specify which compliant crawlers may access content, while recognizing that the rules cannot force malicious bots to comply.
- Separate search access from AI access: Where supported, use the appropriate crawler-specific directives to express your preferences about indexing and AI-related uses of content.
For more details, see the What is Googlebot and how to verify Googlebot requests help documentation.
Forum discussion at X.
