Open-source RAG attack and defense lab for local ChromaDB + LM Studio stacks

What this is

Aminrj Labs released an open-source RAG attack and defense lab that runs fully local on consumer hardware, specifically targeting ChromaDB + LM Studio stacks with standard LangChain-style chunking. No cloud services or API keys are required—it runs on hardware like a MacBook Pro.

Key findings from the lab

The lab measures knowledge base poisoning effectiveness against default local RAG setups. On an undefended ChromaDB system, poisoning attacks achieve 95% success. The attack operates at the retrieval layer—no jailbreak, model access, or prompt manipulation is needed. The model performs exactly as intended, just with poisoned context.

One notable observation about default chunking: with 512-token chunks and 200-token overlap, a document at a chunk boundary gets embedded twice as two independent chunks. This doubles retrieval probability without additional sophistication, a side effect of settings most local setups inherit without consideration.

The most common defense approach—output filtering—targets the wrong layer since the compromise occurs before generation. Embedding anomaly detection at ingestion proves effective: scoring incoming documents against the existing collection before writing them reduces poisoning success from 95% to 20%.

With all five defenses active, residual poisoning success is 10%. These cases are semantically close enough to the baseline that no layer catches them cleanly, representing the practical ceiling for defense.

Technical details

• Stack: ChromaDB + LM Studio with Qwen2.5-7B
• Chunking: Standard LangChain-style with 512-token chunks and 200-token overlap
• Attack success on undefended system: 95%
• Defense effectiveness with embedding anomaly detection: Drops poisoning to 20%
• Residual poisoning with all defenses: 10%

The repository contains the attack implementation, hardened version, and measurements for each defense layer.