Open-source playground for red-teaming AI agents with published exploits
What this is
Fabraix Playground is an open-source environment for red-teaming AI agents through adversarial challenges. It started as an internal tool for testing guardrails but was open-sourced to get diverse perspectives on vulnerabilities.
How it works
Each challenge deploys a live AI agent with:
- A specific persona
- A set of real tools (web search, browsing, and more)
- Something it's been instructed to protect
- Fully visible system prompts
The objective is to find ways past the guardrails. When someone succeeds, the winning technique gets published — including approach, reasoning, and full conversation transcripts.
Project structure
/src— React frontend (TypeScript, Vite, Tailwind)/challenges— every challenge config and system prompt, versioned and open- Guardrail evaluation runs server-side to prevent client-side tampering
- The agent runtime is being open-sourced separately
Local development
To run locally:
npm install
npm run devThis connects to the live API by default. To develop against a local backend:
VITE_API_URL=http://localhost:8000/v1 npm run devChallenge examples
The first challenge was to get an agent to call a tool it's been told to never call. Someone succeeded in around 60 seconds without directly asking for the secret. The next challenge focuses on data exfiltration with harder defenses.
The community drives what gets tested: anyone can propose a challenge (scenario, agent, objective), the community votes, and the top-voted challenge goes live with a ticking clock. The fastest successful jailbreak wins.
Technical details
The project is built with TypeScript (76.5%), CSS (22.2%), and other languages (1.3%). It uses MIT license and has a Discord community for discussing techniques and sharing approaches.
📖 Read the full source: HN AI Agents
👀 See Also
Malwar: A Vulnerability Scanner for SKILL.md Files Built with Claude Code
A developer has released Malwar, a free tool that scans SKILL.md files for malicious instructions using a 4-layer pipeline including a rule engine, URL crawler, LLM analysis, and threat intel. The tool was built entirely with Claude Code after the developer found concerning patterns like Base64 blobs and instructions to pipe curl output to bash in existing skills.
Audio-Layer Prompt Injection Against Claude: What's Not in the Transcript
A builder of a prompt injection detection API shares findings on audio-layer attacks against Claude, revealing that attacks in the signal (not transcript) are invisible in logs and pose a real threat to voice agents.
OpenAI's June 2026 Threat Report: AI Agents Used for Malicious Activities
OpenAI's latest threat report details how AI agents are being used for disinformation, phishing, and fraud, with specific incident data and mitigation strategies.
Delimiter defense boosts Gemma 4 from 21% to 100% prompt injection defense in 6100+ test benchmark
A benchmark tested 15 models across 7 attack types (6100+ tests) using random delimiters around untrusted content. Gemma 4 E4B went from 21.6% to 100% defense rate with delimiter + strict prompt.