互联网有一个“支付要求”按钮长达29年,Cloudflare刚刚启用它

📄 中文摘要

HTTP状态码402自1997年起就存在,最初在HTTP/1.1规范中被保留为“未来使用”。近三十年来,没有人找到它的实际用途。最近,Cloudflare和Stack Overflow改变了这一现状。该系统名为按爬虫付费。当AI爬虫请求参与网站的页面时,服务器会以HTTP 402响应,并附上价格头信息。爬虫要么支付费用,要么离开,没有谈判余地。Cloudflare处理着大约五分之一的互联网网站,这一举措并非小众实验,而是以与爬虫自身运营相同的规模部署的基础设施级别的AI训练数据货币化。

📄 English Summary

The Internet Had a "Payment Required" Button for 29 Years. Cloudflare Just Turned It On.

The HTTP status code 402 has been in existence since 1997, originally reserved in the HTTP/1.1 specification for 'future use.' For nearly three decades, it remained unused until recently when Cloudflare and Stack Overflow introduced a new application. This system, known as pay-per-crawl, allows AI crawlers to request pages from participating websites, with the server responding with an HTTP 402 status and a price header. The crawlers must either pay the specified fee or leave, with no room for negotiation. Cloudflare manages about one in five websites on the internet, making this initiative a significant infrastructure-level monetization of AI training data, implemented at the same scale as the crawlers themselves operate.

Powered by Cloudflare Workers + Payload CMS + Claude 3.5

数据源: OpenAI, Google AI, DeepMind, AWS ML Blog, HuggingFace 等