Custom llama.cpp Backend Offloads LLM Matrix Multiplication to AMD XDNA2 NPU on Ryzen AI MAX 385
Custom Backend for AMD XDNA2 NPU Offload
A developer has created a custom llama.cpp backend that dispatches GEMM operations directly to the AMD XDNA2 NPU on Ryzen AI MAX 385 (Strix Halo). This approach avoids iGPU usage and shared memory contention.
Hardware and Software Configuration
Model: Meta-Llama-3.1-8B-Instruct Q4_K_M
Hardware: Ryzen AI MAX 385, CachyOS 6.19, amdxdna driver, XRT 2.21.75
Performance Results
- Vulkan prefill + NPU decode: 930 t/s prefill (pp512), 43.7 t/s decode (tg64), 41.5W avg power, 0.947 J/tok
- Vulkan only: 833 t/s prefill, 41.6 t/s decode, 52.2W avg power, 1.3 J/tok
- CPU only: 4.6 t/s prefill, 3.76 t/s decode
The NPU decode path saves approximately 10W versus Vulkan-only while matching (and slightly beating) decode throughput, as the iGPU remains free for other work.
Technical Stack
- Kernels: mlir-aie xclbins (Xilinx/mlir-aie, Apache 2.0)
- Runtime dispatch: XRT 2.21.75
- Base: Fork of ggml-org/llama.cpp (MIT)
- Kernel routing: 4 xclbin slots covering different K-dimension tiles with MIN_N/MAX_N routing to select the appropriate kernel at runtime
Performance Ceiling Investigation
The developer attempted to push beyond 43.7 t/s decode with several approaches:
- Batch sweep N=1..64: No improvement (flat performance)
- Int4 double-quant: Killed SNR (44.8 → 19.7 dB) - dead end
- Cascade offload: Ruled out by AMD documentation
- Speculative decoding with Llama-3.2-1B draft: 44% accept rate, 212 t/s draft, but zero effective gain
The lack of improvement from speculative decoding (which normally provides gains with a 44% accept rate) indicates the bottleneck is LPDDR5 bandwidth, not compute. The NPU is already hitting the memory wall, making 43.7 t/s the ceiling for this model on this hardware.
Project Links
- GitHub: https://github.com/BrandedTamarasu-glitch/OllamaAMDNPU
- Changelog: https://brandedtamarasu-glitch.github.io/OllamaAMDNPU/xdna-npu/
The project was built with Claude Sonnet 4.6 / Claude Code, disclosed for reproducibility purposes. The developer is seeking feedback from others running Strix Halo or Phoenix with the amdxdna driver to compare decode throughput on comparable quants and determine if other XDNA2 configurations encounter the same performance ceiling.
📖 Read the full source: r/LocalLLaMA
👀 See Also
PageAgent: Browser AI Agent That Runs Inside Web Pages with Ollama Support
PageAgent is a JavaScript library that runs AI agents directly inside web pages, reading live DOM as text instead of using screenshots. It works with any OpenAI-compatible endpoint including Ollama, enabling local LLM calls directly from the browser.
How I Feel About AI: A Developer's Honest Take on Reasoning, Doom, and the Open Web
A developer reflects on AI: gut-feeling LLMs with emergent reasoning, the threat of superintelligence, and the toll on the open web and artists.
Netflix Releases VOID: Video Object and Interaction Deletion Model on Hugging Face
Netflix has released VOID, a video inpainting model that removes objects from videos along with all physical interactions they induce, including falling objects and displaced items. The model requires a GPU with 40GB+ VRAM and uses quadmask conditioning with two checkpoint files for different refinement levels.
Bio-Inspired Memory System for Local LLMs: LTP and Selective Oblivion Implementation
A developer built a local MCP server implementing bio-inspired memory mechanics including Long-Term Potentiation reinforcement, selective oblivion decay, and weekly consolidation cycles. The system uses hybrid search with sqlite-vec and text fallbacks, non-blocking architecture with asyncio executors, and maintains state via a persistent 'Soul' file.