Benchmark Shows CLI Tool Reduces Claude Code Token Costs by 32% Through Structural Navigation

A developer has open-sourced a CLI tool called Scope that provides Claude Code agents with structural code navigation capabilities, similar to IDE features like "find references" and "go to definition." The tool was built in Rust using tree-sitter and SQLite.

What the Tool Does

The tool gives agents commands like:

• "show me a 180-token summary of this 6,000-token class"
• "search by what code does, not what it's named"

It currently supports TypeScript and C#, with the goal of helping agents navigate code more efficiently than their default grep-based approach.

Benchmark Methodology

The developer ran 54 automated runs on Sonnet 4.6 across a 181-file C# codebase with:

• 6 task categories
• 3 conditions: baseline, tool available, architecture preloaded into CLAUDE.md
• 3 repetitions each

Full NDJSON capture was recorded on every run to decompose tokens into fresh input, cache creation, cache reads, and output. The benchmark runner and telemetry capture are included in the repository.

Key Findings

Contrary to expectations, agents with the tool read more files (6.8 to 9.7 average vs. baseline) but made 67% more code edits per session and finished in fewer turns.

The savings came from shorter conversations, which reduced cache accumulation. Approximately 90% of token cost lives in cache accumulation.

Overall results:

• 32% lower cost per task
• 2x navigation efficiency (nav actions per edit)
• Navigation-to-edit ratio improved from 25:1 (baseline) to 13:1 (with tool) and 12:1 (with architecture preloaded)

Results varied by task type:

• Bug fixes: -62% cost
• New features: -49% cost
• Cross-cutting changes: -46% cost
• Discovery and refactoring tasks: no advantage (baseline agents already navigate these fine)

Important Caveats

The developer notes several limitations:

• p-values don't reach 0.05 at n=6 paired observations (direction is consistent but sample is too small for statistical significance)
• Benchmarked on C# only so far (TypeScript support exists but hasn't been benchmarked yet)
• Cost calculation uses current Sonnet 4.6 API rates: fresh input $3/M, cache write $3.75/M, cache read $0.30/M, output $15/M

The tool is open source and available at github.com/rynhardt-potgieter/scope for developers who want to experiment with improving agent token efficiency.