Cloudflare to Block Mixed-Use AI Crawlers From Ad Pages by Default on Sept. 15, 2026
Updated
Updated · TechCrunch · Jul 1
Cloudflare to Block Mixed-Use AI Crawlers From Ad Pages by Default on Sept. 15, 2026
3 articles · Updated · TechCrunch · Jul 1
Summary
September 15, 2026 is Cloudflare’s deadline for blocking mixed-use crawlers by default on ad-hosting pages unless a site owner changes the setting.
The new default will cover new customers, new sites created by existing customers, and all existing free users, pressing AI firms to separate search bots from training and agent crawlers.
Cloudflare says the shift protects publishers’ intellectual property and bandwidth as bot traffic has already surpassed human traffic online and more than 50% of AI crawl traffic re-fetches unchanged pages.
Google is a clear target: Cloudflare says the largest search engine has access to about 2x more information than rival AI firms, while Google argues its Google Extended bot lets sites opt out of training use without leaving Search.
The policy also ties into Cloudflare’s monetization push, expanding Pay Per Crawl into Pay Per Use with early partners Ceramic.ai and You.com paying publishers when content is used in AI results or premium access.
As websites block AI training, are we heading toward a future of less intelligent and more biased AI assistants?
As tech giants are forced to pay for web data, is the foundational promise of an open internet collapsing?
Cloudflare’s September 2026 AI Crawler Shift: Ending Free Data Access and Launching Pay-Per-Crawl for Publishers
Overview
Cloudflare’s new AI crawler policy, effective September 15, 2026, marks a major shift in how online content is managed and valued. As the traditional 'crawl-for-traffic' model becomes less fair—especially with AI companies using content for training without consent or compensation—Cloudflare is giving publishers more control over their content. This policy addresses the imbalance where companies like Google dominate access and make it hard for publishers to stay visible without giving up their content for AI use. By rebalancing content access and empowering creators, Cloudflare aims to create a more equitable exchange between publishers and AI companies.