SMELT compiler reduces OpenClaw workspace token usage by up to 95%
OpenClaw workspace token optimization tool
SMELT is a Python compiler that processes OpenClaw workspace markdown files to reduce token usage when sending content to AI models like Claude or GPT. The tool addresses a specific inefficiency: OpenClaw resends USER.md, SOUL.md, MEMORY.md, and AGENTS.md on every message, not just at startup.
Performance benchmarks
Testing on a Qwen 3.5 122B model on M3-Ultra hardware revealed:
- Startup bundle: 7,268 tokens reprocessed on every inference call
- 50-message session: Over 350,000 tokens of static workspace files reprocessed
- Query-specific token reductions:
- "Who is Sally?": 1,373 tokens raw → 73 tokens SMELT (94.7% savings)
- "When was John born?": 1,374 tokens raw → 62 tokens SMELT (95.5% savings)
- Broad "Tell me about Alex": 1,373 tokens raw → 328 tokens SMELT (76.1% savings)
- Startup TTFT: 14,121ms raw → 13,273ms SMELT (6% faster)
Technical implementation
SMELT uses a four-layer architecture:
- Archive: Original files are never touched
- Compile: Schema-aware structural compression
- Compress: Dictionary replacement
- Select: Query-conditioned retrieval that only sends relevant records with parent context
The fourth layer (Select) is where the 95% token reduction occurs. The compiler is schema-aware and built specifically for OpenClaw workspace file conventions.
Key findings from development
- Naive JSON conversion (a common optimization attempt) is 30% worse than raw markdown
- Heading stripping provides minimal benefit (7-8% improvement)
- Byte compression and token compression are different - measurements must use the actual tokenizer
- 11 of 13 test files achieved 100% fidelity, with two dense archival files having documented failures
Current limitations and availability
The schema is hand-built for OpenClaw workspace conventions. Support for arbitrary markdown requires schema learning (planned). The tool is free for personal use, with code available on GitHub under TooCas/SMELT and research published on Zenodo with DOI.
The project was built with GPT, Claude, and Codex as collaborators.
📖 Read the full source: r/openclaw
👀 See Also
GPT-5.5 Codex vs Claude Opus 4.7: Real-world coding agent benchmarks
A developer pitted GPT-5.5 Codex against Claude Opus 4.7 on two real tasks: a PR triage bot and a real-time code review UI. Claude shipped cleaner with zero errors; Codex was 18% cheaper but needed a patch pass.
Lightpanda: Open-source headless browser for LLM agents with native MCP server and markdown output
Lightpanda is an open-source headless browser designed for LLM-powered agents that uses 16x less memory than Chrome (215MB vs 2GB) and completes web crawling benchmarks in 5 seconds instead of 47 seconds. It provides native markdown output, semantic tree with interactivity detection, and a built-in MCP server.
ddash: Mermaid Diagram Tool with URL-Based Storage and Claude Code Integration
ddash is a free Mermaid diagram tool where the entire diagram is compressed into the URL hash, requiring no backend, accounts, or storage. It includes a Claude Code skill that lets you generate and open diagrams directly during conversations with commands like /diagram the auth flow.
ANE Optimization Through Phone-Steered AI Experiments Shows Kernel Fusion Benefits
A developer ran 55 experiments on Apple Neural Engine optimization, steering the process from their phone using Claude for brainstorming. Key improvements included fusing 3 ANE kernels into 1 mega-kernel, reducing validation loss from 3.75 to 2.49 and step time from 176ms to 96ms.