One year after launching its one-click 'Block AI Bots' feature, Cloudflare is marking a second 'Content Independence Day' with a major expansion of AI traffic controls. The new tools give every Cloudflare customer — including those on the free tier — the ability to set granular policies for which AI crawlers can access their content, under what conditions, and at what price.
The centerpiece of the announcement is a Pay-Per-Crawl marketplace that lets website owners set per-request pricing for AI training access. When an AI company's crawler hits a participating site, Cloudflare meters the access and handles billing, effectively turning every website into a potential data licensor. Early participants include several major news publishers and educational institutions.
Cloudflare CEO Matthew Prince framed the launch as an evolution of the internet's original bargain. 'For 30 years, the deal was: we crawl you, and you get traffic referrals. AI broke that deal by taking everything and sending nothing back. What we're building is a new deal — one where website owners get compensated for the value their content creates in AI systems.'
The announcement comes as regulators in the EU, UK, and several US states are crafting laws around AI training data rights, and major publishers continue to strike individual licensing deals with AI companies. Cloudflare's marketplace could accelerate that trend by making it technically and commercially trivial for millions of smaller sites to participate.
Source context: Cloudflare Blog
