AgentCrawl Update Adds Critical Crawler Features and Enhancements
The latest update to AgentCrawl enhances its functionality as a TypeScript scraper/crawler, introducing several important features for developers using AI agents. This release focuses on production-readiness by integrating crawler correctness and politeness, caching mechanisms, resumable crawls, and enhanced data extraction capabilities.
Key Details
- Removed Tool Adapters: The update eliminates the tool adapters for the agents SDK and Vercel AI SDK, allowing users to define their tools independently.
- Updated Libraries: The package now includes the latest version of Zod for better data validation.
- Crawler Correctness: Robots.txt compliance is now opt-in and supports Disallow/Allow and Crawl-delay directives. Opt-in sitemap seeding from
/sitemap.xmlis also available. - URL Normalization: Improved URL normalization comprehensively strips tracking parameters and can handle canonical normalization.
- Throttling Options: The crawler supports per-host throttling with configurable
perHostConcurrencyandminDelayMs. - Caching: An opt-in disk HTTP cache for static fetches implements ETag and Last-Modified support. The system caches post-cleaning and markdown conversion of
ScrapedPageand can handle server responses with status 304 by serving cached bodies. - Resumable Crawls: A new opt-in crawlState persistence saves the crawl's frontier, including the queue, visited pages, queued items, errors, and max depth, which allows for resumable crawls without re-visiting pages.
- Data Extraction Improvements: The scraper now supports structured metadata extraction, including Canonical URL, OpenGraph, Twitter cards, and JSON-LD, kept in
metadata.structured. - Chunking for Agents: Opt-in chunking functionality returns
page.chunks[]with an approximate token size, heading path, and citation anchor, which is beneficial for RAG/tool loops.
Who It's For
This update is particularly beneficial for developers utilizing AI agents requiring efficient and structured web scraping capabilities.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Hyper iOS App: Voice Recorder with Real-Time Transcription and Action Extraction
Hyper is an iOS voice recorder app that transcribes conversations in real-time, provides summaries and action items, and allows mid-conversation queries via wakeword detection. It's designed for unstructured meetings like 1:1s, coffee chats, and standups.
Claudetop: Real-Time Cost Monitoring for Claude Code Sessions
Claudetop is an htop-like tool that shows real-time spending, cache efficiency, and model comparisons for Claude Code sessions. It provides slash commands like /claudetop:stats and smart alerts for cost milestones and efficiency issues.
Developer shares CLI tools that work well with Claude Code
A developer switched from MCPs to CLIs for working with Claude Code, finding that Claude handles CLI commands effectively due to training on shell scripts and documentation. They shared specific CLIs they use daily, including gh, ripgrep, stripe, supabase, vercel, sentry-cli, and neon.
ClawTalk iOS App Enables Voice Chat with Self-Hosted OpenClaw AI Agents
ClawTalk is a native iOS app that provides push-to-talk voice chat for OpenClaw self-hosted LLM setups. It features on-device speech-to-text using WhisperKit, real-time streaming responses with markdown rendering, and supports multiple TTS options including ElevenLabs, OpenAI, and Apple's built-in voices.