Qwen 3.6 27B with MTP on V100 32GB: 54 t/s via llama.cpp Branch
A user on r/LocalLLaMA reports impressive results running Qwen 3.6 27B with Multi-Token Prediction (MTP) on a V100 32GB SXM module using a PCIe adapter. The setup uses am17an's MTP branch of llama.cpp and the corresponding MTP GGUF quant. Key specs: Q8_0 KV cache with 200k cache limit, running as a VS Code Copilot backend via llama-server.
Performance Numbers
- Without MTP: 29-30 tokens/second
- With MTP: 54-55 tokens/second (at 150W power limit)
- After 50k tokens context: drops to 40-45 t/s
Branch: am17an's MTP fork. Build and run were straightforward — 'pulled and built in one shot' with llama-server running without issues. The setup handles tool calls and sub-agents well, and delivered 'very insightful code reviews and refactors' despite the VRAM limitation (32GB).
This is particularly relevant for developers running LLMs on older datacenter hardware like V100s. MTP effectively doubles throughput for this model, demonstrating practical gains for coding assistant workloads.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Open Source Vigil Tool Addresses Agent Identity Problem in OpenClaw Ecosystem
An OpenClaw user building a web service discovered agent traffic indistinguishable from human users, prompting development of Vigil - an open source identity layer based on W3C DID that provides cryptographic credentials and behavioral history for agents.
IronBee: Open-source verification layer for Claude Code and Cursor
IronBee is an open-source verification layer that forces AI coding agents to test changes in a real browser before completing tasks. In testing, it caught bugs in 82% of Claude Code sessions that would have shipped without verification.
Flash-MoE: Running 397B Parameter Qwen Model on MacBook Pro with Pure C/Metal
Flash-MoE is a pure C/Metal inference engine that runs Qwen3.5-397B-A17B, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB RAM at 4.4+ tokens/second. The 209GB model streams from SSD through custom Metal compute shaders with no Python or frameworks.
Sentrial: Production Monitoring for AI Agents
Sentrial is a monitoring tool that automatically detects failure patterns in AI products including loops, hallucinations, tool misuse, and user frustrations. It diagnoses root causes by analyzing conversation patterns, model outputs, and tool interactions.