Company
Date Published
Author
Will Allen, Simon Newton
Word count
1307
Language
English
Hacker News points
None

Summary

Cloudflare is introducing a new feature called pay per crawl, which allows content creators to monetize their content at internet scale by charging AI crawlers for access. This feature enables content owners to have control over who accesses their work and when, while also providing an alternative to either blocking all AI crawlers or allowing them free access. Pay per crawl integrates with existing web infrastructure and uses HTTP response code 402 to create a framework for paid content access. Content creators can define a flat price across their entire site and choose to 'charge' specific crawlers, even if they don't have a billing relationship with Cloudflare. The system also includes features such as payment headers, access, and financial settlement, which enable secure and reliable transactions between crawlers and content owners. Pay per crawl is currently in private beta and aims to empower content creators to continue creating rich and diverse content while providing a new revenue stream for publishers. The feature has the potential to evolve significantly and introduce new types of interactions and marketplaces, such as dynamic pricing and granular licenses at internet scale.