Blind Test: 5 Local AI Agents, Same Qwen 27B Model — One Failed an Install Prompt

Someone on r/openclaw ran five local coding agents through an identical five-task suite on the same hardware and model, then hid the names and asked readers to guess which one is OpenClaw. The contenders: Atomic Agent, goose, Hermes, OpenClaw, and OpenCode.

The setup

• Model: Qwen 3.8 27B uncensored, Q4_K_M quantization
• Runtime: llama.cpp
• Hardware: M5 Max MacBook Pro, 64GB RAM
• One agent at a time, auto-approve on
• A script decides pass/fail — no human grading

The tasks were ordinary dev work: merge two contact lists, rename a function across three files, fix a bug with hidden tests, count errors across three log files, and convert a receipt PDF to a spreadsheet.

Results (names hidden)

Agent   1. Contacts   2. Rename   3. Bug fix   4. Errors   5. Receipt   Total
A       ✅ 4:54       ✅ 0:28     ✅ 0:43      ✅ 0:26     ✅ 3:58      10:29
B       ✅ 6:03       ✅ 1:11     ✅ 0:55      ✅ 0:54     ✅ 15:04     24:07
C       ✅ 13:32      ✅ 0:24     ❌ 0:22      ✅ 0:57     ❌ 0:49     16:04
D       ✅ 6:16       ✅ 1:12     ✅ 0:56      ✅ 0:56     ✅ 2:50      12:10
E       ✅ 5:50       ✅ 1:01     ✅ 0:21      ✅ 0:26     ✅ 3:56      11:34

Four of five agents passed all five tasks. Agent C failed two.

Where the agents diverged

On the receipt-to-spreadsheet task, agents A and B hit a missing Python library and installed it without asking. Agent C stopped and asked permission to install — nobody answered, so it failed. Agent D finished the job and only then asked what to name the output.

The failure pattern matters more than the timings: agentic coding tools that block on human input inside an unattended loop are effectively non-functional in this kind of run. C paid for politeness.

On the bug fix, C turned "Q3/Q4" into "q3q4" instead of "q3-q4" — a plausible-looking but wrong transformation that the hidden tests caught.

The other tasks were uniform: all five correctly merged the contact lists to 46 people, all five passed the three-file rename, and all five counted 43 errors across the log files.

Speed spread was wide. E finished in 11:34; B took 24:07, almost entirely due to 15:04 on the receipt task. That is the kind of variance you get from sampling and tool-call retries rather than model quality.

The author says the answer key will be posted in a day or two, so the mapping of letters to products is still open.