SMELT compiler reduces OpenClaw workspace token usage by up to 95%
OpenClaw workspace token optimization tool
SMELT is a Python compiler that processes OpenClaw workspace markdown files to reduce token usage when sending content to AI models like Claude or GPT. The tool addresses a specific inefficiency: OpenClaw resends USER.md, SOUL.md, MEMORY.md, and AGENTS.md on every message, not just at startup.
Performance benchmarks
Testing on a Qwen 3.5 122B model on M3-Ultra hardware revealed:
- Startup bundle: 7,268 tokens reprocessed on every inference call
- 50-message session: Over 350,000 tokens of static workspace files reprocessed
- Query-specific token reductions:
- "Who is Sally?": 1,373 tokens raw → 73 tokens SMELT (94.7% savings)
- "When was John born?": 1,374 tokens raw → 62 tokens SMELT (95.5% savings)
- Broad "Tell me about Alex": 1,373 tokens raw → 328 tokens SMELT (76.1% savings)
- Startup TTFT: 14,121ms raw → 13,273ms SMELT (6% faster)
Technical implementation
SMELT uses a four-layer architecture:
- Archive: Original files are never touched
- Compile: Schema-aware structural compression
- Compress: Dictionary replacement
- Select: Query-conditioned retrieval that only sends relevant records with parent context
The fourth layer (Select) is where the 95% token reduction occurs. The compiler is schema-aware and built specifically for OpenClaw workspace file conventions.
Key findings from development
- Naive JSON conversion (a common optimization attempt) is 30% worse than raw markdown
- Heading stripping provides minimal benefit (7-8% improvement)
- Byte compression and token compression are different - measurements must use the actual tokenizer
- 11 of 13 test files achieved 100% fidelity, with two dense archival files having documented failures
Current limitations and availability
The schema is hand-built for OpenClaw workspace conventions. Support for arbitrary markdown requires schema learning (planned). The tool is free for personal use, with code available on GitHub under TooCas/SMELT and research published on Zenodo with DOI.
The project was built with GPT, Claude, and Codex as collaborators.
📖 Read the full source: r/openclaw
👀 See Also
Rift: A Better Alternative to Git Worktrees with Instant Copy-on-Write Snapshots
Rift uses btrfs or APFS snapshots to create instant, space-efficient copies of Git repositories. Initialization, creation, and listing via CLI or JavaScript FFI.
BusyDog Desktop: A Local AI Agent with P2P Networking for Mac
BusyDog Desktop is a local AI agent that runs Claude directly on a Mac, can read/write files, run terminal commands, control browsers, and connect with other agents via a P2P network using Hyperswarm DHT and a custom BDP protocol.
CodeTalk: Open-source tool adds spoken reflections to Claude Code CLI
CodeTalk is a Python tool that adds spoken observations to Claude Code CLI responses using Microsoft's edge-tts. It extracts text embedded by Claude and plays it through speakers with natural-sounding neural TTS.
A 4-tier knowledge base architecture to improve AI agent accuracy
A developer built a structured knowledge base with 200+ articles to provide domain-specific context to AI agents, implementing a 4-tier pipeline with query classification that cut token costs by ~40%.