Agentic Web Index: AI Bot Traffic Stats Show Mass Scraping, Spoofing

Agentic Web Index: The New AI Bot Traffic Dashboard

Known Agents has published the Agentic Web Index, a dashboard tracking AI agent, crawler, and scraper activity across 5,000+ websites. The data comes from their Agent Analytics and AI Chat Referral Tracking, and it gives a concrete look at what's actually hitting your servers.

Key Metrics: Bots Are 35% of All Traffic

Over the last 90 days, the index reports:

• Bot vs. Human Traffic: 35% of all visits are from bots — down 1% from the previous 90 days.
• Agentrification: 29% of bot traffic is AI-related — up 12%.
• AI Chat Referral Volume: 0.1% of human visits come from AI chat — down 7%.
• Robots.txt Effectiveness: 98.5% of bots follow robots.txt rules.

Top Agent Types and Visitors

When you break down traffic by agent type, traditional search engine crawlers still dominate (22.8%), followed by SEO crawlers (19.5%) and AI search crawlers (12.5%). AI data scrapers like ClaudeBot and Amazonbot make up 11%.

The top five visitors:

1. bingbot — Search Engine Crawler (8.2%)
2. Googlebot — Search Engine Crawler (7.8%)
3. AhrefsBot — SEO Crawler (6.4%)
4. Known Agent DEV — Developer Helper (5.6%)
5. ChatGPT-User — AI Assistant (3.4%)

Other notable entries include ClaudeBot (AI Data Scraper, 3.2%) and meta-externalagent (AI Data Scraper, 2.3%).

Spoofing AI Bots for Mass Vulnerability Scans

Beyond the stats, the site highlights a growing security concern: attackers are spoofing AI agents like ClaudeBot to run mass vulnerability scans. By disguising their traffic as a legitimate AI scraper, they can probe websites for weaknesses while flying under the radar — many sites are unlikely to block a bot they think is just fetching pages for model training.

This is a practical reminder: robots.txt is a suggestion, not a security measure. If you're relying on it to keep scrapers out, you're already exposed. The 98.5% compliance rate only applies to honest bots.

Who This Is For

If you run a website, API, or any public-facing service, this data helps you understand what's actually hitting your infrastructure — and why you need to validate bot identities rather than trust their user-agent strings.