Cloudflare, a company that powers about 20% of all web pages on the web, has announced it is now blocking AI crawlers by default. Plus, it is offering a new model to allow AI services to pay content creators to crawl their content, named Pay Per Crawl.
Cloudflare wrote that they are the "first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default." Now, new customers that sign up for Cloudflare by default will automatically block AI crawlers. Existing customers can block AI crawlers anytime with a single click in their Cloudflare dashboard. This shifts content scraping from an opt-out to opt-in format. There is a lot of buzz on Techmeme on this news.
This site uses Cloudflare extensively, so does 20% of the internet, so this is a big deal. Will I block AI crawlers, probably not - I don't need to get paid for my content here. But most of you probably should (not all of you but many of you).
Pay Per Crawl - This is a new initiative from Cloudflare to let content creators set their rates and get paid by AI services that want to crawl your content. If you want to sign up for Pay Per Crawl, go here to learn more. Publishers can block all AI crawlers, allow specific ones, charge for access, or grant free access — full control over how their content is accessed. AI crawler owners will be able to use Pay Per Crawl to register, see pricing, and choose to pay or not. Pricing will set by both publishers, who can set rates, and AI companies, who can choose whether to access webpages at those rates.
Here are some of the publishers who are part of this announcement: ADWEEK, Atlas Obscura, BuzzFeed, Inc.,Fortune, Stack Overflow, News/Media Alliance, The Atlantic, Battelle Media, Evolve Media, Hyperscience, IAB Tech Lab, O'Reilly Media, Quora, Raptive, Sovrn, Inc., StockTwits, Third Door Media, TIME, Webflow. Yes, Third Door Media is Search Engine Land and Martech (but not this site, I am not blocking anyone...)
Here are some of the settings in Cloudflare:
Here is an awesome video from Cloudflare's CEO on why this is so important for the future of the web:
Forum discussion at X.
