ai-pocalypse Cloudflare has started blocking AI web crawlers by default in a bid to become the internet's gatekeeper.
The term"gatekeeper" has been applied in a pejorative sense to platform companies like Apple and Google that use their contractual and technical control over operating systems to extract monopoly rents from developers within the platform ecosystem.
Cloudflare proposes an alternative interpretation of that role that's less self-serving: to help protect online publishers from the predation of AI firms.
CEO Matthew Prince in a blog post explains that the deal Google made with content creators almost 30 years ago was that the search service would send traffic to websites in exchange for the opportunity to access and index their content.
That deal, he argues, has been broken as Google and rival AI companies deploy search services derived from uncompensated web content crawls, services that starve publishers of revenue by referring less search traffic to their sites.
... the deal that Google made to take content in exchange for sending you traffic just doesn't make sense anymore
"The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you've created, an AI-driven web doesn't reward content creators the way that the old search-driven web did," said Prince."And that means the deal that Google made to take content in exchange for sending you traffic just doesn't make sense anymore."
The web, he said, is being"stripmined by AI crawlers," referring to the automated bots run by companies like Google, Microsoft, OpenAI, and Anthropic, that visit websites and download page content to use for training their AI models.
Recall that last year, Mustafa Suleyman, the CEO of Microsoft AI, argued that content from non-professional publishers is fair game for commercial exploitation."I think that with respect to content that is already on the open web, the social contract of that content since the 1990s has been it is fair use," he said."Anyone can copy it, recreate with it, reproduce with it. That has been freeware, if you like. That's been the understanding."
The courts have also not done much to help publishers, with recent rulings that suggest training on publicly accessible web data qualifies as fair use under copyright law.
In a separate post, Cloudflare's David Belson, head of data insight, and Sam Rhea, VP of product, published data illustrating the disparity between what AI crawlers take and the referral traffic they send back to websites.
During the period between June 19 and 26, 2025, for example,"Anthropic’s AI platform Claude made nearly 71,000 HTML page requests for every HTML page referral," observe Belson and Rhea. We must note that these measures only track traffic from the Claude website, not the app, as the app does not emit a Referer: header. The same goes for the other AI vendors.
That said, the request/referral ratios listed for other AI firms with search operations include OpenAI (1,600:1), Perplexity (202.4:1), Microsoft (40:1), Yandex (18:1), Google (9.4:1), ByteDance (1.4:1), Baidu (1:1), DuckDuckGo (0.3:1), and Mistral (0.1:1).
Pointing to this data, Belson and Rhea contend that legacy search crawlers scanned site content only a few times per visitor sent, and that making a site available for crawling improved revenue. Not so in this new world.
"The new data we are observing suggests that is no longer the case," they write.
In an effort to change that dynamic,"Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content," said Prince.
Large content providers have already arranged payment deals with various AI firms through litigation and contracts. In November 2024, for example, Microsoft entered into an AI training deal with publisher HarperCollins, according to a court filing [PDF] in the copyright lawsuit Bird v. Microsoft. The license gives Microsoft the right to use a given work for AI training for three years, in exchange for a whopping $5,000 payment split between the author and publisher.
Cloudflare aims to make such deals occur through a network handshake. The firm's payment service, now in private beta testing, is called Pay per crawl.
"Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing," explain Will Allen, VP of product, and Simon Newton, engineering manager, in a blog post."Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure."
The amount charged is not clear and is likely to vary by publisher. The payment service will let publishers block AI crawlers, allow specific ones, charge for access, or grant free access.
If publishers see value in Cloudflare's gatekeeping, their adoption of this technology will make the open web less open. But that may be the price necessary to keep the web populated with new material. ®
