FOMOE Enables 397B Qwen3.5 Model Inference on $2,100 Desktop Hardware

What FOMOE Solves
Large Mixture of Experts (MoE) models require hundreds of GBs of weight storage, typically in flash memory like NVMe. During inference, only a small fraction of weights are needed, but you can't predict which ones ahead of time. Random access patterns make flash latencies too high for practical inference on consumer hardware.
How FOMOE Works
The system makes most expert weight reads unnecessary through several techniques:
- Stores the most common experts in GPU memory (VRAM) with an up-to-date rolling expert cache
- Achieves 60% VRAM hit rate with warm start, reducing NVMe reads to 28% (12% served from DRAM)
- Uses dual GPU ping-pong architecture to overlap weight loading and compute
- Implements Cache-Aware Routing (CAR) - when two experts score similarly, the model picks the next-best scoring expert already in VRAM or DRAM cache within acceptable threshold
Performance Results
- 5-9 tokens/second inference speed for Qwen3.5's 397B parameter model
- NVMe reads reduced to 7% with CAR enabled
- Only 3.5% drop in perplexity measured on wikitext
- Hardware requirements: two $500 GPUs, 32GB RAM, one NVMe drive
- Uses Q4_K_M quantization
The implementation consists of approximately 15,000 lines of Claude-driven C/HIP code with heavy human guidance.
📖 Read the full source: r/LocalLLaMA
👀 See Also

Modulus: Cross-repository knowledge orchestration for AI coding agents
Modulus is a desktop app that runs multiple AI coding agents with shared project memory across repositories. It solves cross-repo context problems by letting agents understand dependencies between different codebases without manual explanation.

Recall: Local Project Memory for Claude Code — No Tokens Spent on Summaries
Recall gives Claude Code durable, local session memory via classical summarization. No API key, no external model — context.md is ~1-2K tokens, built offline from session hooks.

2026 Hermes Agent Alternatives Roundup: Self-Hosted Options from OpenClaw to memU Bot
A developer who has been running Hermes since launch tested every self-hosted and managed alternative after the ClawHub security mess. Key findings: OpenClaw (370k stars) but 9 CVEs in 4 days and ~20% malicious packages; TrustClaw rebuilt with OAuth/sandboxing; nanobot at ~4K lines Python with MCP; memU Bot with unique structured memory. Managed options include Perplexity Computer (19 models, $200/mo), Claude Cowork (opens real Mac apps), and KimiClaw (40GB RAG, locked to K2.5, Chinese data law). Full roundup at source.

Open-Source Job Search Framework Built in Claude Code
A developer created an open-source job search framework in Claude Code that handles structured self-profiling, automated job portal scraping, fit evaluation, and a drafter-reviewer agent pipeline for tailored applications. The system stops before submission and requires manual review.