At the end of each year, Cloudflare Radar provides a unique perspective on internet activity worldwide. This year’s report offers a fascinating glimpse into the growing influence of AI bots in shaping how content is consumed, indexed, and referenced, particularly by some of the most prominent AI platforms.
Among the standout data points: Anthropic’s crawler had the highest “crawl-to-refer” ratio among leading AI and search services. But what does that mean for the future of digital visibility, and why should online retailers and marketplaces pay attention?
In simple terms, the crawl-to-refer ratio measures how often a platform’s bots crawl a site versus how often it actually refers users to that site. A high crawl rate but low referral traffic suggests the platform is gathering data, possibly to feed AI models, without sending users back to the original content.
Anthropic’s high crawl-to-refer ratio implies that its AI systems are actively absorbing content to improve understanding, generation, and perhaps even shopping assistance capabilities, without necessarily directing users back to source websites.
This presents both opportunities and challenges for e-commerce players.
Traditionally, search engines indexed product pages and sent shoppers to merchant websites. Now, AI agents might skip that last step, delivering summarized or repurposed product information directly inside chat interfaces or smart assistants.
For example, imagine a shopper asking an AI assistant to recommend a “lightweight 14-inch laptop under €800.” Instead of linking to retailer pages, the AI could generate an answer from crawled data by comparing specs, prices, and reviews based on content it has already ingested.
This shift signals a growing need for well-structured, machine-readable product content. If your catalog isn’t easily parsed or enriched, you might be left out of AI-driven shopping journeys entirely.
At Icecat, we’ve long understood the value of structured, standardized product information. Our global product content syndication network supports over 60 languages and 24 million product titles, with data optimized for both human and machine consumption.
Our content is not only visible in webshops and marketplaces, but also indexed by AI systems that now shape consumer journeys behind the scenes. As AI platforms like Anthropic, OpenAI, and Google continue to expand their web activity, having enriched, machine-readable product data becomes a competitive necessity.
The insights from Cloudflare suggest that AI crawlers are becoming some of the most active visitors to websites globally. But unlike human users, they don’t shop; but they train.
Soon, AI agents may act as personal shopping assistants, making decisions or recommendations on users’ behalf. When that happens, your ability to influence those recommendations will depend on the quality and accessibility of your data.
Now is the time for e-commerce companies to ask:
Anthropic’s top-ranking crawl-to-refer ratio is not just a technical detail. It’s a signal that AI-driven content consumption is changing the rules of online visibility. For e-commerce retailers and brands, adapting to this new environment is critical to staying relevant, not just in search results, but also in AI responses.
As AI becomes a gatekeeper in the product discovery journey, your content strategy must evolve to meet both human and machine expectations.
Read further: News, AI, Anthropic, Cloudfare, e-commerce, ecommerce, Icecat
