Custom llama.cpp Backend Offloads LLM Matrix Multiplication to AMD XDNA2 NPU on Ryzen AI MAX 385

✍️ OpenClawRadar📅 Published: March 26, 2026🔗 Source
Custom llama.cpp Backend Offloads LLM Matrix Multiplication to AMD XDNA2 NPU on Ryzen AI MAX 385
Ad

Custom Backend for AMD XDNA2 NPU Offload

A developer has created a custom llama.cpp backend that dispatches GEMM operations directly to the AMD XDNA2 NPU on Ryzen AI MAX 385 (Strix Halo). This approach avoids iGPU usage and shared memory contention.

Hardware and Software Configuration

Model: Meta-Llama-3.1-8B-Instruct Q4_K_M

Hardware: Ryzen AI MAX 385, CachyOS 6.19, amdxdna driver, XRT 2.21.75

Performance Results

  • Vulkan prefill + NPU decode: 930 t/s prefill (pp512), 43.7 t/s decode (tg64), 41.5W avg power, 0.947 J/tok
  • Vulkan only: 833 t/s prefill, 41.6 t/s decode, 52.2W avg power, 1.3 J/tok
  • CPU only: 4.6 t/s prefill, 3.76 t/s decode

The NPU decode path saves approximately 10W versus Vulkan-only while matching (and slightly beating) decode throughput, as the iGPU remains free for other work.

Technical Stack

  • Kernels: mlir-aie xclbins (Xilinx/mlir-aie, Apache 2.0)
  • Runtime dispatch: XRT 2.21.75
  • Base: Fork of ggml-org/llama.cpp (MIT)
  • Kernel routing: 4 xclbin slots covering different K-dimension tiles with MIN_N/MAX_N routing to select the appropriate kernel at runtime
Ad

Performance Ceiling Investigation

The developer attempted to push beyond 43.7 t/s decode with several approaches:

  • Batch sweep N=1..64: No improvement (flat performance)
  • Int4 double-quant: Killed SNR (44.8 → 19.7 dB) - dead end
  • Cascade offload: Ruled out by AMD documentation
  • Speculative decoding with Llama-3.2-1B draft: 44% accept rate, 212 t/s draft, but zero effective gain

The lack of improvement from speculative decoding (which normally provides gains with a 44% accept rate) indicates the bottleneck is LPDDR5 bandwidth, not compute. The NPU is already hitting the memory wall, making 43.7 t/s the ceiling for this model on this hardware.

Project Links

  • GitHub: https://github.com/BrandedTamarasu-glitch/OllamaAMDNPU
  • Changelog: https://brandedtamarasu-glitch.github.io/OllamaAMDNPU/xdna-npu/

The project was built with Claude Sonnet 4.6 / Claude Code, disclosed for reproducibility purposes. The developer is seeking feedback from others running Strix Halo or Phoenix with the amdxdna driver to compare decode throughput on comparable quants and determine if other XDNA2 configurations encounter the same performance ceiling.

📖 Read the full source: r/LocalLLaMA

Ad

👀 See Also

Argus: A GitHub App That Reviews CLAUDE.md Files and Posts Scores on PRs
Tools

Argus: A GitHub App That Reviews CLAUDE.md Files and Posts Scores on PRs

Argus is a GitHub App built with Claude Code that reviews CLAUDE.md files and posts a score on every pull request. After testing on multiple repositories, the most common failures are missing explicit scope limits and escalation paths.

OpenClawRadar
Open-Source Claude IDE Bridge Connects Dispatch, Desktop App, and Claude Code
Tools

Open-Source Claude IDE Bridge Connects Dispatch, Desktop App, and Claude Code

The claude-ide-bridge is an MIT-licensed open-source tool that connects Claude Code to your IDE, providing access to LSP, debugger, terminals, git, and GitHub through 124 tools. It enables a workflow where tasks sent via Dispatch from a phone are handled by the Claude desktop app, which uses Claude Code to write code and run tests while interacting with the IDE.

OpenClawRadar
Nia-docs tool creates local filesystem from documentation URLs for Claude AI
Tools

Nia-docs tool creates local filesystem from documentation URLs for Claude AI

The nia-docs tool lets you run npx nia-docs with a documentation URL to create a local filesystem of the docs, which Claude AI can then access directly without additional configuration.

OpenClawRadar
Skales Desktop AI Agent Built with Claude, Features Clippy-Style Mascot
Tools

Skales Desktop AI Agent Built with Claude, Features Clippy-Style Mascot

Skales is a desktop AI agent that runs locally on Windows and macOS, using Claude via OpenRouter/Anthropic API for reasoning and tool execution. It includes a floating Desktop Buddy mascot with a paperclip skin reference and can execute commands like sending emails, managing files, browsing the web, and managing calendars.

OpenClawRadar