IETF Groups Push 2 Standards to Curb Web Crawling as AI Fears Threaten Open Access
Updated
Updated · EFF · Jun 17
IETF Groups Push 2 Standards to Curb Web Crawling as AI Fears Threaten Open Access
1 articles · Updated · EFF · Jun 17
Summary
Two IETF working groups are advancing standards that would let websites more easily restrict automated access to public web content, a shift critics say could weaken the open internet.
AI Preferences would add robots.txt-style signals against AI-related crawling, while Web Bot Auth is exploring cryptographic bot identification that could help sites block or license only approved crawlers.
The push is being driven by publishers' and platforms' fears that AI bots strain infrastructure and divert traffic, advertising and licensing revenue from source websites.
EFF argues the proposals could spill beyond abusive AI scraping to lawful uses by journalists, researchers, archivists, watchdogs, accessibility tools and startups that cannot pay for access.
The group said it and allied open-internet advocates have already helped stop some of the most restrictive ideas, framing the fight as a broader battle over whether internet standards stay neutral or become gatekeeping tools.
As AI's data appetite grows, is the era of the free and open internet coming to an end?
Will new web standards create a paywall for public data, blocking researchers and startups?
When bots can be cryptographically verified, who becomes the ultimate gatekeeper of online information?
From Open Web to Paywalls: How IETF and AI Crawling Standards Could Reshape Internet Access
Overview
The Internet Engineering Task Force (IETF) is tackling the challenges of AI web crawling by focusing on better communication between content publishers and AI model trainers. After the IAB AI-CONTROL Workshop highlighted the urgent need for improved protocols, the IETF formed two key groups: the AI Preferences (AIPREF) Working Group and the Web Bot Authentication (webbotauth) Working Group. AIPREF aims to help publishers and AI vendors clearly express and understand AI usage preferences, leading to the publication of a new vocabulary draft. These efforts mark important steps toward managing how AI interacts with web content.