Building a Fully Local Multi-Agent Assistant with OpenClaw and Ollama

A developer on r/openclaw is building a fully local personal AI assistant (think Jarvis) using OpenClaw as the agent framework, Ollama for local inference, and a MINISFORUM AI X1 with Ryzen AI 9 HX470, 96GB RAM, and 2TB NVMe (GPU via Oculink planned). The goal: a proactive multi-agent system that integrates smart home, documents, calendar, health, and communications — all locally, with no sensitive data leaving the infrastructure.

Stack Details

• Agent Framework: OpenClaw
• Inference Engine: Ollama
• Models: qwen3.5:35b-a3b (main), gemma3:4b (home), mistral:7b (life/gmail)
• MCP Servers: Home Assistant, Gmail
• Interface: Telegram Bot, future STT integration into smart home

Sub-Agent Architecture

The main routing agent delegates to specialized sub-agents:

• HA Agent – smart home control and debugging (started)
• Gmail Agent – email management (started)
• Life Agent – calendar, to-do, grocery list management (tbb)
• Health Agent – health and sport data monitoring (tbb)
• Research Agent – web + document RAG (in paperless ngx on NAS) (tbb)
• Dev Agent – coding tasks with separate coding, testing, doc agents (tbb)

Challenges & Open Questions

• Context bloat: Context size grows very large even for simple messages. Configuration likely needs better MCP server scoping and sub-agent tool isolation.
• MCP per-agent scoping: No native way to restrict MCP servers to specific agents yet. Seeking recommended workarounds given an open bug.
• Sub-agent config: Looking for a well-structured agents.list example for this multi-agent setup.
• Local model selection: Reliable tool-calling with Ollama under 32GB VRAM — any recommendations?
• Inference environment: Considering switching to llama.cpp instead of Ollama if it provides better control.

The developer is open to feedback on approach and configuration. If you've tackled similar multi-agent scaling issues or have MCP scoping workarounds, join the discussion on Reddit.