Tired of all of those “smart ass AI crawlers” meddling with your content? Well, it’s going to happen regardless, but at least you get to harvest a bit of financial benefit through Cloudflare’s new Pay per Crawl feature.
Up until now, content creators basically had two options: let AI bots scrape everything for free, or block them entirely. But Cloudflare thinks there should be a third way – let them in, but only if they pay.
Utilizing an “ancient tech” in the form of the HTTP Response Code 402 “Payment Required” (Perhaps you’ve seen this when you need to access some research paper back in the college days), Pay per Crawl simply replies with a 402 response towards the AI crawler, if the website admin enabled monetization.
Unless the bot agrees to pay for it, the official response should come with the payment details and proper header, which is then unlocked for it.
Cloudflare isn’t just letting publishers toggle a paywall on and off, but with giving full control. Apparently, the settings menu is quite expansive and you can create exception lists to allow some bots in for free (maybe your own AI crawler? I don’t know), block others entirely, or charge a flat per-request rate across your site. And even if a crawler isn’t set up for billing yet, you can still “charge” it, which works like a soft block that says, “pay up if you want in.”
On the technical side, there’s a big focus on making sure bots can’t fake who they are. That’s where Web Bot Auth comes in, as each crawler needs a cryptographic key pair and has to register with Cloudflare to send authenticated requests. Headers like [crawler-exact-price] or [crawler-max-price] let bots indicate what they’re willing to pay, and if it matches the publisher’s rate, the content gets served.
While this sounds like the beginning of the “AI Vaccine” era, Pay per Crawl is still in private beta now, but it could create a good example and baby steps towards properly monetizing content for AI access. If you’re super interested and want to try it out, Cloudflare is taking signups here.
