The Future of Content and AI: Pay per Crawl and What's Next
#ThisWeekInNET — Episode 99
In this episode, host João Tomé is joined by Will Allen, Cloudflare’s VP of Product Management, to discuss Pay-Per-Crawl and our new permission-based model for AI bots. These updates, launched on Content Independence Day, aim to reshape how AI models access and reward content, shifting from opt-out to opt-in.
They explore how AI Overviews are changing the old “traffic for content” model, and how Cloudflare is helping creators take control through tools like AI Audit. Plus: the future of trustworthy content, bot authentication, and the rise of a fairer content economy.
⏱️ Timestamps:
00:02 - The problem with current content monetization and AI
03:57 - Matthew Prince on Google's changing model and Content Independence Day
08:28 - Will Allen's role at Cloudflare and the intersection of creators and AI
11:11 - What is Content Independence Day and Pay-Per-Crawl?
13:42 - The opportunity for content creators in the AI age
14:30 - Imagining a pay-per-crawl future for deep research agents
17:16 - Content creators' control and AI Audit tool
19:54 - Publisher and AI company participation in Pay-Per-Crawl
22:05 - Why AI companies want good content
23:33 - Trust in content and digital provenance
26:09 - The relationship between AI companies and publishers
28:28 - Evolution of robots.txt and managed controls
31:09 - Crawlers' impact on traffic and attribution
34:12 - Feedback and next steps for content independence
37:03 - Bot authentication and cryptographic verification
39:37 - What's coming next for Pay-Per-Crawl and related tools
40:42 - How to connect with Will Allen and provide feedback
Check the mentioned blog posts here:
https://blog.cloudflare.com/tag/pay-per-crawl/
🔗 Watch more episodes on our show page: https://cloudflare.tv/ThisWeekInNET
🔗 https://www.youtube.com/playlist
#ThisWeekInNET #Cloudflare #ContentIndependence #PayPerCrawl #AI #Creators