Automated QA and Testing with AI: A New Era for Software Testing

Antirez, creator of Redis, outlines a practical method for using LLM agents to automate QA and testing. The approach: create a markdown file that instructs an AI agent to act as a QA engineer, performing manual testing on a new release.

How It Works

The markdown file includes:

• Instructions to check new commits since the last release.
• Specific QA tasks, like distributed inference testing or speed regression checks.
• SSH endpoints, keys, and paths for integration tests.

The agent inspects the changes and identifies what could be affected, then runs a specialized QA pass targeting regressions.

Example: DwarfStar Inference Engine

For DwarfStar, an open-weight LLM inference engine, antirez uses this file to:

• Distributed inference test: Runs across two MacBooks, checking output coherence and GGUF file support on both machines.
• Speed regression check: No need to specify previous speeds — the agent learns dynamically from the codebase.
• Integration verification: Covers complex setups that are hard to automate traditionally.

Example: Redis Arrays

For Redis Arrays, the agent builds a large array-based Redis application, sets up production replication with persistence, simulates days of usage with many users, and flags anomalies.

Psychological QA

The agent also reviews features for clarity and documentation: identifies features that look surprising, undocumented, or sloppy from a user perspective. This catches UX issues that manual QA normally skips.