Local RAG Tool Built with Nemotron Nano 9B v2 and vLLM Tool Calling
Technical Implementation Details
A developer has shared their approach to building a local-first RAG research tool that runs entirely on a single GPU. The entire backend is contained in a single app.py file.
Stack and Configuration
The tool uses Nemotron Nano 9B v2 Japanese on vLLM with FP16 quantization, running on an RTX 5090 GPU. The backend combines FastAPI + SQLite FTS5 + Jinja2. For tool calling, the developer uses NVIDIA's official parser plugins, specifically --tool-call-parser nemotron_json and --tool-parser-plugin, noting that Nemotron v2 requires custom parser plugins rather than the built-in vLLM parsers (which are for v3).
Key Design Decisions
The system implements an extract → execute two-step flow:
- When a question is asked, the system first extracts bilingual keywords (English and Japanese) via LLM
- Runs FTS5 search on local sources AND DuckDuckGo web search in parallel
- Shows results with checkboxes for user selection
- Only after user selection does it generate the final response
This approach avoids dumping 100k+ tokens of context and hoping the model figures it out.
Performance and Features
- Tool Calling: The model autonomously decides when to search the web, working surprisingly well at temperature 0.1
- Prefix Cache Warmup: Instead of caching everything at source load, the KV cache is warmed up when the user sees the source preview. By the time they click Execute, the prefix is already cached using
--enable-prefix-cachingon vLLM - Bilingual FTS5 Search: User query → Nemotron extracts keywords in both English and Japanese → OR-joined FTS5 MATCH query, effective for multilingual patent/research data
Performance Numbers
- ~80-120 tok/s output
- 8192 max tokens
- Source extraction: ~3-5s (keyword extraction + FTS5 + DDG parallel)
- Full response with 5 sources + 3 web results: ~50s for a detailed answer on RTX 5090
Setup and Source
The source code is available at https://github.com/soy-tuber/SoyLM. It's a single file application that can be installed with uv pip install -r requirements.txt. Note that it requires vLLM with the Nemotron parser plugins separately.
📖 Read the full source: r/LocalLLaMA
👀 See Also
ClawTalk iOS App Enables Voice Chat with Self-Hosted OpenClaw AI Agents
ClawTalk is a native iOS app that provides push-to-talk voice chat for OpenClaw self-hosted LLM setups. It features on-device speech-to-text using WhisperKit, real-time streaming responses with markdown rendering, and supports multiple TTS options including ElevenLabs, OpenAI, and Apple's built-in voices.
HomeButler: Zero-token homelab management for OpenClaw agents
HomeButler is a single Go binary that lets OpenClaw agents manage homelab infrastructure without API keys or tokens. It runs locally and keeps all operations on your network.
LLM Cost Profiler: Open-source tool tracks API spending to make case for local models
LLM Cost Profiler is a Python tool that tracks every API call to OpenAI/Anthropic, showing exactly what you're spending and where. It exposes tasks that are overpriced relative to their complexity, providing concrete dollar amounts to justify moving to local models.
Tendr Skill Adds CLI-Based Long-Term Memory with Hierarchy to Reduce Token Usage
A new OpenClaw skill separates reasoning from execution for long-term memory operations, using a CLI tool to handle structural changes deterministically. It supports wikilinks and explicit semantic hierarchy across files to reduce token consumption and prevent error accumulation.