Open-weight models under 100GB can't beat Claude Haiku on coding benchmarks
A recent analysis of open-weight language models reveals a significant performance gap compared to Anthropic's Claude Haiku on coding benchmarks. The comparison was conducted using specific testing parameters and memory requirements.
Benchmark methodology
The evaluation compared models on two coding benchmarks: LiveBench (January 2026) and Arena Code/WebDev. Testing was performed against Claude Haiku 4.5 with thinking capabilities enabled. Models were plotted according to memory requirements for local deployment.
Technical specifications
- Quantization: Q4_K_M
- Context length: 32K
- KV cache: q8_0
- VRAM estimation: Calculated using the author's custom calculator
Key findings
No open-weight model under 100GB of memory comes close to Claude Haiku's performance on either benchmark. The nearest competitor is Minimax M2.5, which requires approximately 136GB of memory and roughly matches Haiku's performance on both benchmarks.
The analysis highlights the current gap between proprietary and open-weight models in the under-100GB category for coding tasks. The author expresses frustration with this limitation and calls for development of smaller models that could at least match Haiku's capabilities.
📖 Read the full source: r/LocalLLaMA
👀 See Also
MiniMax Releases MaxClaw: Cloud-Hosted AI Agent Based on OpenClaw
MiniMax has launched MaxClaw, a fully managed cloud-hosted AI agent built on the OpenClaw framework. It deploys in 10 seconds without Docker or servers and features the MiniMax M2.5 model with 229B parameters, 200K-1M token context, and up to 100 tokens/s inference speed.
Why OpenClaw's Open Source Architecture Matters
Coding Agents Supersede Human Code Review: Paper Argues Traditional Review Is Dead
arXiv paper argues coding agents have crossed the threshold to replace human code review, offering lower cost and higher throughput.
Local LLM Benchmark: Backend Generation by Function Calling – GLM, Qwen, DeepSeek Compared
A rigorous benchmark of local and frontier LLMs for backend code generation via function calling, with scoring rubric. Key findings: qwen3.5-35b-a3b matches gpt-5.4 on DB/API design, and dense Qwen 27B beats 397B MoE. Frontier models dropped due to cost.