On 1 July, Cloudflare used its second annual “Content Independence Day” to quietly redraw the terms of trade for everyone who publishes anything on the internet. From 15 September, the company’s default settings will block AI crawlers that mix search, training, and agent duties from any page that carries advertising. At the same time, its year-old Pay Per Crawl experiment is evolving into something more consequential: Pay Per Use, a model that pays publishers when their content actually shapes an AI answer, not merely when a bot fetches the page.
If you run marketing for a business that depends on organic reach, this is not infrastructure news. It is a repricing event. The assumption baked into every content strategy of the last two years (publish openly, get ingested by AI, hope the answer engines send you customers) just acquired a tollbooth, a meter, and a deadline.
What actually changed on 1 July
Cloudflare sits in front of more than 20% of the web, which means its defaults function less like product settings and more like policy. The company now sorts crawler traffic into three buckets: search bots that index for discovery, training bots that harvest content to build models, and agent bots that fetch pages on behalf of an AI completing a task. Website owners can allow, block, or block-on-ad-pages each category independently.
The deadline matters more than the interface. From 15 September, “mixed-use” crawlers (the ones that refuse to separate search from training and agent duties) get blocked by default on any ad-supported page. The new defaults apply to new Cloudflare customers, new sites from existing customers, and every free-tier customer who has not changed their settings. The starting position of the open web has flipped: content that earns money is now off-limits to AI unless its owner opts in.
The numbers behind the decision explain the urgency. By June 2026, bots accounted for 57.4% of all web traffic to HTML content on Cloudflare’s network, with humans making up the remaining 42.6%. Crawling for AI training alone represents just over half of total traffic, while classic search crawling has shrunk to roughly 11%. A year and a half ago, training accounted for around 22% of crawler requests. It is now 52% and climbing. Announcing the change, Cloudflare CEO Matthew Prince said the company “must go further and act faster so that a sustainable ecosystem can emerge.”
For marketers, the translation is simple: the majority of your website’s visitors are no longer people, and the fastest-growing category of visitor extracts value without ever sending a customer back.
From Pay Per Crawl to Pay Per Use: the unit of value moved
Last year’s Pay Per Crawl was a blunt instrument with an elegant mechanism. It revived HTTP 402 (“Payment Required”), a response code that sat dormant in the web’s plumbing for three decades, and let publishers set a price per fetch: allow, charge, or block.
The problem is that a crawl is a terrible proxy for value. A page can be fetched once and cited in thousands of AI answers, or fetched ten thousand times and never surface anywhere. Cloudflare’s own data shows more than half of all crawl traffic from legitimate bots goes toward re-fetching pages that have not changed since the last visit. AI companies burn compute retrieving stale content; publishers pay bandwidth to serve it. Both sides lose.
Pay Per Use moves the meter from the fetch to the outcome. Two partners are live at launch. Ceramic.ai runs a pay-per-query model: publishers who opt in are paid every time their content appears in Ceramic’s search results. You.com pays on demand when its agents access a specific piece of premium content at the moment it is needed, with no upfront commitment on either side. Cloudflare has also opened a waitlist for a Monetization Gateway that will let site owners charge for any page, dataset, or API, settled in stablecoins over the open x402 protocol.
Read those mechanics again and notice what your content has become: inventory. Metered, priced per use, with reporting attached. That is a fundamentally different asset class from “marketing collateral,” and it deserves a fundamentally different management approach.
The attribution problem your dashboards don’t show yet
Every marketing leader has watched organic click-through rates sag and struggled to prove why. The external data is now unambiguous. Pew Research found that when Google shows an AI summary, users click a traditional result just 8% of the time, versus 15% without one, and click a source inside the AI summary in only 1% of cases. Ahrefs’ February 2026 analysis put the click reduction from AI Overviews at 58% for top-ranking results. Seer Interactive measured organic click-through on AI Overview queries down 61% in fifteen months, with paid click-through down 68% over the same window.
Meanwhile the extraction ratios remain absurd. At the point Cloudflare published figures in mid-2025, Anthropic’s crawler was fetching 38,000 pages for every one visit it referred back; OpenAI’s ratio stood at 1,091 to one. Some of the most heavily crawled site categories have seen human traffic fall as much as 40% in under a year, which is why many publishers now plan openly for “Google Zero”: a world where search referrals approach nil.
Here is the genuinely new part. The Pay Per Use reporting stack gives publishers, for the first time, query-level attribution inside AI answers: which queries surfaced your content, the exact page and snippet shown, and your average ranking position within the engine’s output. That is the raw material of a real discipline the industry has started calling answer engine optimisation (AEO), the AI-era heir to SEO. Tooling is arriving fast; HubSpot launched a dedicated AEO product at $50 a month (€44 / R820) after disclosing that its own customers’ organic traffic had fallen 27% year on year.
The practical move for a CMO is to start instrumenting three numbers now, before budget season: crawl volume by intent (who is taking what, and for search, training, or agents), citation share (how often your brand appears inside AI answers for your priority queries), and answer-referral rate (what an AI citation is actually worth in visits and pipeline). You cannot negotiate, or optimise, what you have never measured.
The coming split: cited content versus licensed content
Once content is metered, a portfolio decision follows. Every content asset you own will end up on one of two tracks.
Cited content stays open on purpose. Its job is to be ingested, surfaced, and attributed inside AI answers: your point-of-view pieces, comparison pages, documentation, and anything that wins when a prospect’s first touchpoint is a chatbot’s recommendation. This is the track where AEO replaces SEO as the operating discipline, and where being quotable, structured, and fresh beats being merely comprehensive.
Licensed content gets a price. Proprietary research, benchmark data, expert analysis, anything an answer engine needs but cannot cheaply reproduce, moves behind the meter and earns per use. This is no longer theoretical: more than 50 publisher-AI licensing agreements have been signed since 2023, and the infrastructure to do it programmatically (rather than through bespoke lawyer-heavy deals) is exactly what Cloudflare shipped this month. The strategic question stops being “should our content be free?” and becomes “which content is advertising, and which content is product?”
The Google complication
One player scrambles this neat picture. Google still drives roughly 88% of referral traffic, and its flagship Googlebot remains a mixed-use crawler: the same fetch feeds Search, AI Overviews, and AI Mode. Cloudflare estimates this gives Google access to about twice the information available to rival AI companies, because publishers cannot stay visible in Search without also feeding Google’s AI experiences. Blocking Googlebot is commercial self-harm; allowing it means the split above only applies to everyone except the biggest answer engine. Regulators have noticed (the UK is forcing Google to let publishers opt out of AI search results without losing ranking), but until that unbundling lands globally, every content strategy needs a Google exception written into it.
What to do before 15 September
The deadline is seven weeks out. None of the following requires waiting for it.
First, audit your bot traffic. If your site sits behind Cloudflare, the AI Crawl Control and new attribution dashboards will show you who is crawling, how often, and how little they refer back. If it does not, your log files will. Most marketing teams have never seen their own crawl-to-referral ratio, and it tends to concentrate the mind.
Second, set a deliberate crawler posture rather than inheriting a default. Decide, per content section, whether you are open to search, training, and agents, and whether ad-supported pages should carry different rules. “Whatever the defaults happen to be” is now a strategy with revenue consequences, in both directions.
Third, tier your content into cited and licensed candidates. Most of your library belongs on the open, cited track. The 5% that is genuinely proprietary is about to have a market price for the first time. Know which is which before someone else prices it for you.
Fourth, stand up AEO measurement next to your SEO reporting. Citation share and answer-referral rate will be board-slide metrics within a year, and the teams that started tracking in 2026 will own the baseline.
Fifth, re-stress-test your channel mix. If organic clicks on AI-answered queries are down by half or more and paid click-through is falling alongside, the honest response is not to squeeze the same channels harder. It is to price in a structurally smaller open web and rebalance toward the channels you control: email, community, first-party data, and the CRM systems that turn a smaller volume of visits into a larger share of revenue.
Be clear-eyed about the limits. Pay Per Use is an experiment with two named partners, pricing undisclosed. Licensing income today is bespoke and will not replace lost referral revenue for most businesses any time soon. But the direction is set, and the leverage has moved. For a decade, the deal was content for traffic, and the traffic side of that deal has been quietly defaulting. As of 15 September, the defaults finally push back.
Your content was always an asset. Now it has a meter, a price sheet, and a market. Businesses that manage it that way (with measurement, tiering, and deliberate distribution economics) will fund their marketing from both sides of the tollbooth. Businesses that keep publishing everything, open, for free, for everyone, are donating inventory to their competitors’ answer engines.
Blaque Software helps businesses build marketing systems for exactly this shift: content and SEO strategy that survives the answer-engine era, CRM and lifecycle architecture that compounds owned audiences, and measurement frameworks that prove what every channel is actually worth. If your growth still depends on traffic someone else controls, let’s talk.