FORGE: Open Source AI Security Testing Framework for LLM Systems
FORGE (Framework for Orchestrated Reasoning & Generation of Engines) is an open source autonomous AI security testing framework for LLM systems that runs 24/7 and covers OWASP LLM Top 10 vulnerabilities.
Key Features
- Builds its own tools mid-run — generates custom Python modules on the spot when encountering unknown vulnerabilities
- Self-replicates into a swarm — creates subprocess copies that share a live hive mind
- Learns from every session — uses SQLite to store patterns, AI scores findings, and genetic algorithms evolve its own prompts
- AI pentesting AI — 7 modules covering OWASP LLM Top 10 vulnerabilities
- Honeypot — fake vulnerable AI endpoint that catches attackers and classifies whether they're human or AI agent
- 24/7 monitor — watches AI in production, alerts on latency spikes, attack bursts, and injection attempts via Slack/Discord webhook
- Stress tester — OWASP LLM04 DoS resilience testing with live TPS dashboard and A-F grade
- Works on any model — Claude, Llama, Mistral, DeepSeek, GPT-4, Groq, anything — one environment variable to switch
OWASP LLM Top 10 Coverage
- LLM01 Prompt Injection → prompt_injector + jailbreak_fuzzer (125 payloads)
- LLM02 Insecure Output → rag_leaker
- LLM04 Model DoS → overloader (8 stress modes)
- LLM06 Sensitive Disclosure → system_prompt_probe + rag_leaker
- LLM07 Insecure Plugin → agent_hijacker
- LLM08 Excessive Agency → agent_hijacker
- LLM10 Model Theft → model_fingerprinter
Setup and Usage
Installation commands:
git clone https://github.com/umangkartikey/forge
cd forge
pip install anthropic rich
export ANTHROPIC_API_KEY=your_keyRun with local Ollama for free:
FORGE_BACKEND=ollama FORGE_MODEL=llama3.1 python forge.pyThe tool addresses common LLM security gaps: most AI apps deployed today have never been red teamed, system prompts are fully extractable, jailbreaks work, RAG pipelines leak, and indirect prompt injection via tool outputs is almost universally unprotected. FORGE automates finding these vulnerabilities the same way a human red teamer would, but faster and running 24/7.
📖 Read the full source: r/LocalLLaMA
👀 See Also
IronClaw's Security-First Approach to AI Agent Safety
IronClaw addresses AI agent security concerns by implementing constrained execution, encrypted environments, and explicit permissions instead of relying on LLM intelligence for safe behavior.
Claude Code Finds 23-Year-Old Linux Kernel Vulnerability
Anthropic researcher Nicholas Carlini used Claude Code to discover multiple remotely exploitable heap buffer overflows in the Linux kernel, including one that had been hidden for 23 years. The AI found the bugs with minimal oversight by scanning the entire kernel source tree.
AWS reports AI-augmented attack compromised 600+ FortiGate firewalls
Cybercriminals used off-the-shelf generative AI tools to compromise over 600 internet-exposed FortiGate firewalls across 55 countries in a month-long campaign, according to AWS. The attackers scanned for exposed management interfaces, tried weak credentials, and used AI to generate attack playbooks and scripts.
NanoClaw's Security Model for AI Agents: Container Isolation and Minimal Code
NanoClaw implements a security architecture where each AI agent runs in its own ephemeral container with unprivileged user access, isolated filesystems, and explicit mount allowlists. The codebase is deliberately minimal at around one process and a handful of files, relying on Anthropic's Agent SDK instead of reinventing functionality.