OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.
Ryan Siddle from Merj wrote on LinkedIn, "OpenAI scaling up their infrastructure ahead of Thanksgiving & Black Friday with a lot of /28 blocks."
He added later in the comments, "That’s just across OpenAI User for new IPs. It doesn’t include what they already had. We’ve seen quite a significant ramp up over the past 1-2 months."
"Going back to what I said earlier today, make sure these are added into your WAF rules (or use CDN bot detection)," he reminded.
Here is his screenshot:
Yay - more crawlers on our sites.
Forum discussion at LinkedIn.
