Categories: Web and IT News

Cloudflare blocks AI crawlers by default, adds “Pay Per Crawl”

Cloudflare activated a sweeping change to the way AI companies access the web. Every new domain that signs up to its network now blocks unverified AI crawlers by default. This affects an estimated 20% of the entire Internet.

Site owners are prompted up-front to decide whether to allow bot access. They can also charge for it or flat-out refuse it. Cloudflare CEO Matthew Prince framed this as “put[ting] the power back in the hands of creators”.

Beyond “Allow” or “Deny”: meet HTTP 402

Cloudflare’s parallel launch of Pay Per Crawl lets publishers price each bot request. When an AI agent hits a pay-walled URL, it receives an HTTP 402 Payment Required response. This response advertises the cost. A bot can then retry the request with a signed payment header to complete the transaction. Alternatively, it can back off if the price is too steep.

Cloudflare acts as merchant-of-record and enforces the rule set after its existing WAF and bot-management layers.

Why the clamp-down?

Internal telemetry published on Cloudflare Radar helps explain the urgency. During the week of 19–26 June, Anthropic’s Claude crawler made around 70,900 HTML requests for each referral click-through. Meanwhile, OpenAI’s GPTBot had a ratio that remained in the thousands-to-one range.

Put bluntly: most generative-AI systems slurp content on the web without sending readers or revenue back.

Publishers cheer, but keep one hand on the steering wheel

Media groups were quick to endorse the move. DMGT vice-chair Rich Caccappolo called the default block “a structured and transparent relationship between content creators and AI platforms”.

News/Media Alliance president Danielle Coffey likewise praised the framework for creating “a more equitable exchange” between the two industries.

AI crawlers are still growing fast

Even as Cloudflare rolls out stricter controls, overall crawler traffic continues to swell. Company data show a year-over-year rise of 18% in combined AI-and-search crawling. GPTBot volume increased by 305% in the same period. Googlebot also grew by 96% as Google leans into AI Overviews.

Developers react: opportunity ≠ headlock

A lively Hacker News thread shows developers split: some hail Pay Per Crawl as a long-needed revenue lane, while independent builders fear roadblocks for open-source search projects or archival crawls. Some have expressed concerns that this scheme could tax innovation before it starts.

What happens next?

  • For publishers: The default block is live today for new Cloudflare domains; existing customers can opt-in from the dashboard.
  • For AI firms: Authenticating crawlers, and perhaps negotiating per-URL pricing, becomes table stakes for large-scale training.
  • For regulators: Cloudflare’s protocol could crystallize around its proposed Web Bot Auth standard, giving lawmakers a technical hook for future policy.

Whether the market embraces HTTP 402 micro-payments or gravitates toward negotiated bulk licenses, Cloudflare has lobbed a high-profile gauntlet: “scrape-first, apologize-later” is no longer the default on one-fifth of the web.

Cloudflare blocks AI crawlers by default, adds “Pay Per Crawl” first appeared on Web and IT News.

awnewsor

Recent Posts

New Era Medical Continues to Support Clearwater Residents With Ozone Therapy in Clearwater, FL

Dr. Jared L. Roscoe, DC, IMD, discusses ozone therapy with an adult patient during a…

43 minutes ago

Wolfeboro Clinic Highlights Cold Laser and Class IV Options for Joint and Soft Tissue Pain

A healthcare specialist at Healing Frontiers administers non-invasive cold laser therapy in Wolfeboro, NH, to…

43 minutes ago

LEVEL8 Black Friday: A Guide to the Grace and Luminous Carry-Ons

Berlin – Nov. 20, 2026 – LEVEL8 has announced a 20% Black Friday discount on…

43 minutes ago

Metaguest Reports Second Consecutive Quarter of Positive EBITDA and Announces AGM Results

table, td, td p, span, font {font-family: Arial, Helvetica, sans-serif ; font-size:11pt ;} View this…

44 minutes ago

17 Cory Watson Attorneys Named to 2026 Mid-South Super Lawyers

Birmingham law firm Cory Watson Attorneys has 17 lawyers named to the 2026 Mid-South Super…

44 minutes ago

Zoho Unveils Built-In Travel Capabilities in Zoho Spend to Help Businesses Access Negotiated Rates and Enforce Policy at Booking

Built for mid-market and enterprise: Zoho Spend offers industry-first capabilities that combine travel, expense, procurement,…

44 minutes ago

This website uses cookies.