SMELT compiler reduces OpenClaw workspace token usage by up to 95%
OpenClaw workspace token optimization tool
SMELT is a Python compiler that processes OpenClaw workspace markdown files to reduce token usage when sending content to AI models like Claude or GPT. The tool addresses a specific inefficiency: OpenClaw resends USER.md, SOUL.md, MEMORY.md, and AGENTS.md on every message, not just at startup.
Performance benchmarks
Testing on a Qwen 3.5 122B model on M3-Ultra hardware revealed:
- Startup bundle: 7,268 tokens reprocessed on every inference call
- 50-message session: Over 350,000 tokens of static workspace files reprocessed
- Query-specific token reductions:
- "Who is Sally?": 1,373 tokens raw → 73 tokens SMELT (94.7% savings)
- "When was John born?": 1,374 tokens raw → 62 tokens SMELT (95.5% savings)
- Broad "Tell me about Alex": 1,373 tokens raw → 328 tokens SMELT (76.1% savings)
- Startup TTFT: 14,121ms raw → 13,273ms SMELT (6% faster)
Technical implementation
SMELT uses a four-layer architecture:
- Archive: Original files are never touched
- Compile: Schema-aware structural compression
- Compress: Dictionary replacement
- Select: Query-conditioned retrieval that only sends relevant records with parent context
The fourth layer (Select) is where the 95% token reduction occurs. The compiler is schema-aware and built specifically for OpenClaw workspace file conventions.
Key findings from development
- Naive JSON conversion (a common optimization attempt) is 30% worse than raw markdown
- Heading stripping provides minimal benefit (7-8% improvement)
- Byte compression and token compression are different - measurements must use the actual tokenizer
- 11 of 13 test files achieved 100% fidelity, with two dense archival files having documented failures
Current limitations and availability
The schema is hand-built for OpenClaw workspace conventions. Support for arbitrary markdown requires schema learning (planned). The tool is free for personal use, with code available on GitHub under TooCas/SMELT and research published on Zenodo with DOI.
The project was built with GPT, Claude, and Codex as collaborators.
📖 Read the full source: r/openclaw
👀 See Also
OpenClaw Setup on Ubuntu UTM VM with LLM API and Ollama Access
A user successfully configured OpenClaw in a sandboxed Ubuntu VM on an M3 Mac, with access to both local Ollama on macOS and external LLM APIs like Gemini, Claude, and DeepSeek. Sample configuration files and troubleshooting notes are available on GitHub.
Zikra: Self-Hosted MCP Memory Server for Claude Code, Cursor, and Codex
Zikra is a self-hosted MCP memory server that automatically saves every decision, error, and requirement when Claude Code sessions end via a Stop hook, creating a shared memory pool accessible across tools and team members.
Open Source Curated Collection of OpenClaw Resources Unveiled
Discover a new open-source collection of OpenClaw resources, curated by the community to enhance AI development and collaboration.
NLA Transforms Gemma 3’s Internal Activations into Readable Text for Any Token
Anthropic released Natural Language Autoencoders (NLA) that decode a model’s internal state into text. Paired with Gemma 3, the Auto Verbalizer explains what the model was “thinking” for any generated token. Weights are on Hugging Face; demo on Neuronpedia.