Benchmark Results for Small Local and OpenRouter Models on Agentic Text-to-SQL Task
A developer has published benchmark results for small local and OpenRouter models on an agentic text-to-SQL task. The benchmark takes English queries like "Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory" and converts them to SQL that is tested against database tables.
Benchmark Details
The agent can see query results and modify SQL to fix issues, with a limit on debugging rounds. The benchmark is deliberately short with 25 questions and runs in much less than 5 minutes for most models, making it practical for testing different configurations. It's designed to be tough enough to separate the best models from others.
Key Findings
- The best open models identified were kimi-k2.5, Qwen 3.5 397B-A17B, and Qwen 3.5 27B
- NVIDIA Nemotron-Cascade-2-30B-A3B outscores Qwen 3.5-35B-A3B and matches Codex 5.3
- Mimo v2 Flash was described as "a gem of a model"
Self-Hosted Option
The benchmark now includes the ability to run it yourself against your own server using the WASM version of Llama.cpp. The developer is seeking feedback on what to change for version 2 and wants to see scores others get with different configurations.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Customizing Claude AI for Improved Feedback
Adjust Claude AI's settings to avoid excessive agreement and push for more critical thinking and practical feedback.
Mesh LLM: Distributed AI Computing on Iroh – Run LLMs Across Your Own GPUs
Mesh LLM pools GPUs you already have across machines and exposes them as one OpenAI-compatible API. Distributes models locally, via peer routing, or as a pipeline across several nodes using iroh's QUIC transport.
Claude-Skills Maintainer Seeks Feedback on 181 Agent Skills Library
Reza, maintainer of claude-skills, is asking the community for feedback on his open-source library containing 181 agent skills, 250 Python tools, and 15 agent personas that work across 11 AI coding tools. He's questioning whether the isolated skill approach is effective and wants input on missing skills, persona-based agents, and tool integrations.
Deploy OpenClaw on VPS with a One-Command CLI
A Reddit user claims to have developed a CLI that deploys OpenClaw on a $4.99/month VPS with a single command, offering a cost-effective alternative to using Mac Minis.