Agentic Web Index: AI Bot Traffic Stats Show Mass Scraping, Spoofing
Agentic Web Index: The New AI Bot Traffic Dashboard
Known Agents has published the Agentic Web Index, a dashboard tracking AI agent, crawler, and scraper activity across 5,000+ websites. The data comes from their Agent Analytics and AI Chat Referral Tracking, and it gives a concrete look at what's actually hitting your servers.
Key Metrics: Bots Are 35% of All Traffic
Over the last 90 days, the index reports:
- Bot vs. Human Traffic: 35% of all visits are from bots — down 1% from the previous 90 days.
- Agentrification: 29% of bot traffic is AI-related — up 12%.
- AI Chat Referral Volume: 0.1% of human visits come from AI chat — down 7%.
- Robots.txt Effectiveness: 98.5% of bots follow robots.txt rules.
Top Agent Types and Visitors
When you break down traffic by agent type, traditional search engine crawlers still dominate (22.8%), followed by SEO crawlers (19.5%) and AI search crawlers (12.5%). AI data scrapers like ClaudeBot and Amazonbot make up 11%.
The top five visitors:
bingbot— Search Engine Crawler (8.2%)Googlebot— Search Engine Crawler (7.8%)AhrefsBot— SEO Crawler (6.4%)Known Agent DEV— Developer Helper (5.6%)ChatGPT-User— AI Assistant (3.4%)
Other notable entries include ClaudeBot (AI Data Scraper, 3.2%) and meta-externalagent (AI Data Scraper, 2.3%).
Spoofing AI Bots for Mass Vulnerability Scans
Beyond the stats, the site highlights a growing security concern: attackers are spoofing AI agents like ClaudeBot to run mass vulnerability scans. By disguising their traffic as a legitimate AI scraper, they can probe websites for weaknesses while flying under the radar — many sites are unlikely to block a bot they think is just fetching pages for model training.
This is a practical reminder: robots.txt is a suggestion, not a security measure. If you're relying on it to keep scrapers out, you're already exposed. The 98.5% compliance rate only applies to honest bots.
Who This Is For
If you run a website, API, or any public-facing service, this data helps you understand what's actually hitting your infrastructure — and why you need to validate bot identities rather than trust their user-agent strings.
📖 Read the full source: HN AI Agents
👀 See Also
OpenClaw: Four Critical Issues Developers Need to Know
From image-passthrough bugs to dead channel adapters, lock file visibility, and missing parallel coordination — four production-impacting issues from the OpenClaw repo (366k stars).
Claude Artifacts API Usage Counts Against Chat Quota, Not API Billing
Using Claude artifacts within Claude makes normal API calls that are intercepted by Anthropic and authenticated through the logged-in session, counting against a plan's chat quota rather than API billing. Users can verify this by testing artifacts and checking that API usage remains at zero in the Claude Console.
Uber Burned Through Its Yearly Claude Code Budget in 4 Months — Here's What That Means
Uber reportedly exhausted its entire Claude Code budget for the year by month four. The post breaks down why usage patterns broke the subscription model and what devs can learn about setting boundaries with AI coding agents.
Claude MAX Plan Now Includes 1M Token Context Window at No Extra Cost
The Claude MAX plan has been automatically upgraded to include a 1 million token context window without additional API-based usage charges, with users reporting significantly reduced token usage and elimination of context window management overhead.