Benchmark Results: GitHub CLI vs MCP Approaches for AI Agents

Benchmark Results: GitHub CLI vs MCP Approaches

A Reddit user conducted an independent study comparing different methods for exposing GitHub tools to AI agents. The benchmark tested four approaches: GitHub CLI, MCP (Model Context Protocol), MCP with Tool Search, and MCP with Code Mode, using real data and practical tasks.

Key Findings

• GitHub MCP is 2–3x more expensive to use than GitHub CLI. The source notes there's "almost no practical reason to use their MCP except for some of the different handling of security."
• Tool Search saves upfront tokens but spends them on extra turns. Whether this trade-off pays depends on task complexity. Tool Search also introduces a new failure mode due to imperfect search accuracy.
• Code Mode is the cheapest way to use MCP, but still 2x more expensive than CLI, and it's very slow. Code Mode introduces a unique failure mode when the agent writes buggy code or poor error handling.
• The benchmark suggests it's possible to push CLIs further toward higher success rates at lowest cost and latency with a principled design approach that treats agent ergonomics as a first-class concern.

Open Source Resources

The author has detailed their approach at https://axi.md and open-sourced the benchmark harness, results, and reference implementation of gh-axi at https://github.com/kunchenguid/axi.