BotCost.dev: Free Analyzer to See How Much AI Bots Cost Your Site
BotCost.dev is a free analyzer that lets you paste your Nginx, Apache, Cloudflare, or Vercel log file and immediately see which AI bots are crawling your site and exactly how much bandwidth they're burning. No account, no upload — everything stays in your browser.
How it works
- Drop your log file — supports access.log, access.csv, or vercel-logs.json. All processing is client-side.
- Analyze — the tool matches each request against 18 known bot fingerprints including GPTBot, ClaudeBot, Bytespider, CCBot, and Perplexity. It calculates the estimated bandwidth cost in dollars.
- Block — outputs a ready-to-paste WAF rule for Cloudflare, Nginx, Next.js, or a robots.txt snippet. Copy and deploy instantly.
Key numbers from the source
- 38% of all web traffic is now non-human, mostly AI scrapers.
- For a typical site with 50k visitors/month, AI bots cost an estimated $180/month in bandwidth.
- BotCost verification latency is <10ms, running at Cloudflare's edge with zero impact on real users.
Why this matters
If you run a site with public content, bots like GPTBot and ClaudeBot can rack up significant bandwidth costs. This tool gives you a concrete dollar figure so you can decide whether to block them. The one-line WAF rule approach means you can stop the bleeding in minutes without complex configuration.
📖 Read the full source: HN AI Agents
👀 See Also
CK Search: Local Semantic Search Tool with MCP Server Integration
CK Search is a local semantic search tool with a built-in MCP server that indexes any text directory without cloud dependencies. The tool can be used by AI agents via MCP, and the source provides a practical walkthrough covering setup, strengths, and limitations compared to grep.
Browser39: A Headless Web Browser for AI Agents
Browser39 is a headless web browser designed specifically for AI agents that converts web pages to token-optimized Markdown locally, runs JavaScript, manages cookies and sessions, queries the DOM, and fills forms. It's a single binary with no external browser needed, no fees, and no external service.
Hollow AgentOS: Run Claude-like agents locally on RTX 5070 using Qwen 3.5 9B
A self-modifying agent system running Qwen 3.5 9B on local hardware cuts Claude API costs by 50%. Uses iterative testing and self-improvement loop to develop software without human intervention.
Auto-optimize: A Claude Code Plugin for Autonomous Performance Optimization
A developer built auto-optimize, a Claude Code plugin that autonomously runs profile → plan → benchmark loops to optimize code performance. In one test, it achieved a 27% faster hash table across all benchmark scenarios in about 3 hours.