Qwen 3.6 27B with MTP on V100 32GB: 54 t/s via llama.cpp Branch
A user on r/LocalLLaMA reports impressive results running Qwen 3.6 27B with Multi-Token Prediction (MTP) on a V100 32GB SXM module using a PCIe adapter. The setup uses am17an's MTP branch of llama.cpp and the corresponding MTP GGUF quant. Key specs: Q8_0 KV cache with 200k cache limit, running as a VS Code Copilot backend via llama-server.
Performance Numbers
- Without MTP: 29-30 tokens/second
- With MTP: 54-55 tokens/second (at 150W power limit)
- After 50k tokens context: drops to 40-45 t/s
Branch: am17an's MTP fork. Build and run were straightforward — 'pulled and built in one shot' with llama-server running without issues. The setup handles tool calls and sub-agents well, and delivered 'very insightful code reviews and refactors' despite the VRAM limitation (32GB).
This is particularly relevant for developers running LLMs on older datacenter hardware like V100s. MTP effectively doubles throughput for this model, demonstrating practical gains for coding assistant workloads.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Banker Creates Credit Due Diligence Tool with 31 AI Prompts Using Only Claude
A banker with 17 years of MSME credit underwriting experience in India has open-sourced 31 AI prompts that transform a visiting card into a full credit due diligence report, compressing a 3-4 week process to 30 minutes. The tool was built entirely through conversation with Claude, without writing any code.
OpenAlly: Local AI Assistant for Android with Phone Control
OpenAlly is an Android app that runs an AI assistant locally on your phone via an embedded Node.js process, with 51 built-in skills and phone control capabilities through Aster companion. It connects to 19+ messaging platforms and supports 18 model providers with your own API keys.
Lightpanda: Open-source headless browser for LLM agents with native MCP server and markdown output
Lightpanda is an open-source headless browser designed for LLM-powered agents that uses 16x less memory than Chrome (215MB vs 2GB) and completes web crawling benchmarks in 5 seconds instead of 47 seconds. It provides native markdown output, semantic tree with interactivity detection, and a built-in MCP server.
OpenClawDreams: A Dream Simulator Extension for OpenClaw Agents
OpenClawDreams is an extension that adds a background reflection process and nightly dream cycle to OpenClaw agents. It captures encrypted conversation summaries to a local SQLite database, processes them during background cycles, and generates consolidated insights that get pushed into the agent's persistent memory.