WebClaw: Open-Source MCP Server for Web Extraction with Claude
WebClaw is an MCP server built in Rust that adds web extraction capabilities to Claude Desktop and Claude Code. It addresses the problem where Claude's built-in web_fetch gets blocked on most real websites, returning 403 Forbidden errors, Cloudflare challenges, or empty responses.
Technical Solution
The server uses TLS fingerprinting at the HTTP layer so websites see a real Chrome browser fingerprint instead of a bot. In testing against 10 popular sites, Claude's built-in web_fetch failed on all 10, while WebClaw successfully extracted content from 9 out of 10 sites.
Features
scrape: Extract clean content from any URLcrawl: Recursive site crawlingextract: Structured data extraction using JSON schema or natural language promptssummarize: Page summariesbrand: Extract colors, fonts, logos from any sitediff: Track content changesmap,batch,search,researchtools
Claude Code Development
The extraction pipeline was implemented with Claude Code, including:
- Scoring algorithm based on text density, semantic tags, and link ratio penalties
- Noise filter that strips navigation, ads, and cookie banners without false positives on Tailwind classes
- Multiple rounds of refinement for edge cases
Setup and Usage
Setup requires one command:
npx create-webclawThe tool detects Claude Desktop and Claude Code automatically and writes the configuration. No API key is needed for 8 of the 10 tools, and everything runs locally.
Performance Benefits
The output is optimized for Claude's context window. A typical news article goes from 4,820 tokens (raw HTML) to 1,590 tokens in WebClaw's LLM format - a 67% reduction while maintaining the same content.
WebClaw is free and open source under the MIT license, available at https://github.com/0xMassi/webclaw.
📖 Read the full source: r/ClaudeAI
👀 See Also
MCP Support Merged into llama.cpp with New WebUI Features
The Model Context Protocol (MCP) pull request for llama.cpp has been merged, adding MCP support, tool calls, an agentic loop, and a server selector to the llama-server/WebUI side.
Agent Browser Protocol: Open-source Chrome fork for AI agents achieves 90% on Mind2Web benchmark
Agent Browser Protocol (ABP) is an open-source Chrome fork that freezes JavaScript and time after each action to convert web browsing into multimodal chat for AI agents. It achieved 90.53% on the Online Mind2Web Benchmark and can be added to Claude Code with a single command.
HolyClaude: Docker Container for Claude Code with Browser UI and Headless Chromium
HolyClaude is an open-source Docker container that packages Claude Code CLI with a browser UI, headless Chromium, and additional AI coding tools. Setup requires only docker compose up and provides access at localhost:3001.
Orion: Bypassing CoreML to Run and Train LLMs Directly on Apple Neural Engine
Orion is an open-source Objective-C system that bypasses Apple's CoreML to run and train LLMs directly on the Apple Neural Engine (ANE), achieving 170+ tokens/s for GPT-2 124M decode and stable multi-step training on a 110M parameter transformer.