Cloudflare is taking another step toward reshaping how AI companies interact with online content by announcing a broad set of new tools aimed at giving website owners more control, visibility, and revenue opportunities as AI-driven web traffic continues to grow.
With more than 50% of today’s internet traffic now coming from bots and automated agents rather than humans, AI is now the primary way users discover information. While many publishers want their content to appear in AI-powered search and answer engines, they also want greater control over how that content is used, particularly for AI training or commercial applications.
To address this, Cloudflare is introducing new AI crawler classifications, enhanced analytics, and commercial partnerships intended to create a more transparent ecosystem for both publishers and AI companies. The initiative focuses on three areas: allowing site owners to decide how AI systems access their content, providing deeper insight into AI traffic, and creating new ways for creators to be compensated when AI services benefit from their work.
New Crawler Policy
Slated for a September 15 rollout, newly added websites and Cloudflare customers with pages containing advertisements will, by default, allow AI search indexing while blocking AI training and agent-related crawling. AI crawlers that combine search, training, and agent functions without giving publishers granular control will also be blocked on ad-supported pages by default. Existing free-tier customers that have not customized their settings will automatically receive the same defaults on that date, although site owners can modify these policies at any time through the Cloudflare dashboard.
The introduction is not a one-time but rather an ongoing process, as Cloudflare will be spending the next 2 months gathering feedback from website operators and AI companies while testing the new classifications. The company argues that separating search, AI agents, and model training creates a fairer environment for publishers while encouraging AI companies to be more transparent about how their crawlers operate.
Attribution Business Insights Dashboard
This new feature allows organizations to verify and monitor how AI bots interact with their websites with detailed visibility. Beyond traditional security metrics, business teams will be able to monitor which AI companies crawl their content, how frequently it is accessed, and how much referral traffic AI-generated answers send back to their sites. The company also sees this as laying the groundwork for Answer Engine Optimization (AEO), an emerging practice similar to SEO that focuses on how content is cited and surfaced within AI-generated responses rather than conventional search rankings.
Other Nitty Gritty Improvements
- AI crawling reduction – Only crawl when it’s necessary; Old web pages with no changes in a given period will not be crawled
- Pay Per Crawl – Now called Pay Per Use; Publishers are paid when content is used to generate value, such as appearing in AI-generated answers or premium AI search results
