Anthropic Bots Crawl Web 2,800-to-1 Against Referrals, Undermining Sites’ Traffic Model
Updated
Updated · Business Insider · Jul 9
Anthropic Bots Crawl Web 2,800-to-1 Against Referrals, Undermining Sites’ Traffic Model
1 articles · Updated · Business Insider · Jul 9
Summary
Cloudflare’s July 1-7 data showed Anthropic’s bots requested webpage crawls about 2,800 times for every one referral sent back to publishers, the widest imbalance among major AI firms it tracked.
That ratio measures how often AI bots take content versus how often their services return users to source sites — a proxy for whether AI companies still support the web’s long-standing traffic-for-crawling bargain.
Anthropic’s figure improved from roughly 8,800-to-1 in early April and 24,700-to-1 in early May, but Cloudflare’s ranking still put OpenAI and Perplexity behind it, with Microsoft and Google further back.
DuckDuckGo came closest to a balanced exchange at about 3 crawls per referral, underscoring how far most AI answer engines have shifted toward extracting content while sending little traffic back.
Anthropic has disputed Cloudflare’s methodology and said newer search features are increasing referrals, but the broader concern is that direct AI answers could weaken publishers’ incentives to keep producing high-quality web content.