Benchmark shows context engine reduces AI coding agent costs by 3x on SWE-bench
A developer benchmarked four AI coding agents on SWE-bench Verified using the same Claude Opus 4.5 model, with context management as the only variable. The results show significant cost differences for similar performance levels.
Benchmark setup
The test used a 100-task stratified subset of SWE-bench Verified with all 12 repositories represented proportionally. All agents ran Claude Opus 4.5 with the same $3/task budget and 250-turn limit. The only difference was the context layer in front of the model.
Results
- Context engine + Claude Code: 73.0% Pass@1, $0.67/task
- Live-SWE-Agent: 72.0% Pass@1, $0.86/task
- OpenHands: 70.0% Pass@1, $1.77/task
- Sonar Foundation: 70.0% Pass@1, $1.98/task
The most expensive setup costs 3x more per task for a lower resolution rate. Eight tasks were solved only by the setup with the context layer - bugs that the model couldn't fix without seeing the right code.
Limitations
On matplotlib (rendering-heavy, visual output code), the context engine scored 43% while Sonar Foundation hit 86%. Graph-based context is less effective when relevant code doesn't follow dependency chains.
How the context layer works
Instead of letting Claude read entire files, it pre-indexes the codebase into a dependency graph using tree-sitter + SQLite (30 languages supported) and returns a ranked context capsule: full source for functions that matter, skeletonized signatures for everything connected to them. The agent starts every task already knowing what's relevant.
It includes session memory that persists across sessions via MCP. When code changes, previous observations get flagged as stale automatically, so the agent doesn't re-explore the same things.
The system is 100% local with no cloud, no account, and no code leaving your machine. It works with Claude Code and 11 other agents via MCP.
Open source availability
The benchmark harness, all evaluation logs, per-instance results, and comparison scripts are available on GitHub at github.com/Vexp-ai/vexp-swe-bench. The tool itself is available at vexp.dev with a free tier, VS Code extension, or CLI. Full benchmark results with charts are at vexp.dev/benchmark.
📖 Read the full source: r/ClaudeAI
👀 See Also
Open-source markdown vault gives Claude persistent memory across sessions
My Portable Brain is a markdown vault structure with an agent runtime layer that provides Claude with persistent context about identity, projects, goals, CRM, and weekly plans. It works natively with Claude Code and Claude Cowork, uses plain markdown files, and runs background scripts nightly to keep context fresh.
Hyper iOS App: Voice Recorder with Real-Time Transcription and Action Extraction
Hyper is an iOS voice recorder app that transcribes conversations in real-time, provides summaries and action items, and allows mid-conversation queries via wakeword detection. It's designed for unstructured meetings like 1:1s, coffee chats, and standups.
Foreman: Open Source Slack Bot for Remote Control of Local Claude Code
Foreman is a free, open source Slack bot that provides remote control for locally running Claude Code instances. It allows developers to send tasks to Claude from their phone while maintaining full local access to filesystem, tools, and environment.
OpenClaw Agent Relay Plugin Fixes Telegram Delivery in Multi-Agent Setups
The openclaw-agent-relay plugin addresses the persistent issue where sessions_send responses go to webchat instead of Telegram by using gateway WebSocket RPC to trigger agent turns with deliver:true, eliminating the need for workarounds like explicit message tools or announce steps.