AgentCrawl Update Adds Critical Crawler Features and Enhancements
The latest update to AgentCrawl enhances its functionality as a TypeScript scraper/crawler, introducing several important features for developers using AI agents. This release focuses on production-readiness by integrating crawler correctness and politeness, caching mechanisms, resumable crawls, and enhanced data extraction capabilities.
Key Details
- Removed Tool Adapters: The update eliminates the tool adapters for the agents SDK and Vercel AI SDK, allowing users to define their tools independently.
- Updated Libraries: The package now includes the latest version of Zod for better data validation.
- Crawler Correctness: Robots.txt compliance is now opt-in and supports Disallow/Allow and Crawl-delay directives. Opt-in sitemap seeding from
/sitemap.xmlis also available. - URL Normalization: Improved URL normalization comprehensively strips tracking parameters and can handle canonical normalization.
- Throttling Options: The crawler supports per-host throttling with configurable
perHostConcurrencyandminDelayMs. - Caching: An opt-in disk HTTP cache for static fetches implements ETag and Last-Modified support. The system caches post-cleaning and markdown conversion of
ScrapedPageand can handle server responses with status 304 by serving cached bodies. - Resumable Crawls: A new opt-in crawlState persistence saves the crawl's frontier, including the queue, visited pages, queued items, errors, and max depth, which allows for resumable crawls without re-visiting pages.
- Data Extraction Improvements: The scraper now supports structured metadata extraction, including Canonical URL, OpenGraph, Twitter cards, and JSON-LD, kept in
metadata.structured. - Chunking for Agents: Opt-in chunking functionality returns
page.chunks[]with an approximate token size, heading path, and citation anchor, which is beneficial for RAG/tool loops.
Who It's For
This update is particularly beneficial for developers utilizing AI agents requiring efficient and structured web scraping capabilities.
📖 Read the full source: r/LocalLLaMA
👀 See Also
HolyCode: Docker Container for Persistent Claude AI Coding Environments
HolyCode is a Docker container that maintains AI coding environment state across machine switches and rebuilds. It includes 30+ preinstalled tools, browser automation with Chromium + xvfb + Playwright, and preserves context in ./data/opencode.
2026 Hermes Agent Alternatives Roundup: Self-Hosted Options from OpenClaw to memU Bot
A developer who has been running Hermes since launch tested every self-hosted and managed alternative after the ClawHub security mess. Key findings: OpenClaw (370k stars) but 9 CVEs in 4 days and ~20% malicious packages; TrustClaw rebuilt with OAuth/sandboxing; nanobot at ~4K lines Python with MCP; memU Bot with unique structured memory. Managed options include Perplexity Computer (19 models, $200/mo), Claude Cowork (opens real Mac apps), and KimiClaw (40GB RAG, locked to K2.5, Chinese data law). Full roundup at source.
3D-Printed Clawd Mascot with ESP32-Powered Mochi Bot
A developer built a physical 3D Clawd inspired by the Claude Code mascot, with an ESP32-driven Mochi bot featuring a small display. Files and code available on MakerWorld and GitHub.
Open-sourced Claude Code prompts reverse-engineered using Claude
A developer used Claude to rewrite all 26 prompts from Claude Code's source after studying the TypeScript codebase during a brief public availability window. The MIT-licensed collection includes system, tool, agent, memory, coordinator, and utility prompts.