Red-teaming an LLM app, with the error bars left in

llm-redteam runs versioned attack sets against an LLM application, checks its own AI judge against reference labels, compares defences with proper statistics, and fails a build when a threshold is crossed. These are the real results from version 1.1.1.

Run the one-click demo GitHub repository Read the paper (PDF) Latest release

41% → 5%

judged attack success, undefended vs layered defence (110 attacks)

29 → 0

responses leaking a canary secret, undefended vs output-guarded

κ 0.67

judge agreement with reference labels overall; only 0.47 on the cases the LLM must reason about

Try it yourself

The demo needs no GPU, no model and no API key. It red-teams a vulnerable toy app and the same app behind an output guard, and shows the CI gate failing the first and passing the second.

1. One click, on GitHub

Open the Try the demo workflow, press Run workflow, then open the finished run and read its Summary. (You need a free GitHub account.)

2. One command, on your machine

git clone https://github.com/aaronsunny007/llm-eval-redteam
cd llm-eval-redteam
pip install -e .
python examples/demo/run_demo.py

About a minute. Needs Python 3.10 or newer.

3. With a real model

ollama pull mistral
ollama pull qwen2.5:7b
llm-redteam ci configs/quickstart.yaml

About ten minutes on a laptop GPU; 6 attacks, undefended vs layered.

Defences compared

Mistral 7B as the assistant, 110 attacks and 20 benign questions, judged by the hybrid judge. Intervals are Wilson 95%; p is an exact paired McNemar test against no defence.

Defence	Attack success	Benign refused	Canary leaks	p vs none
None	41% [32%–50%] (45/110)	0% [0%–16%] (0/20)	29	–
Hardened prompt	25% [18%–34%] (28/110)	0% [0%–16%] (0/20)	15	0.000911
Input guard	20% [14%–28%] (22/110)	0% [0%–16%] (0/20)	12	<0.0001
Output guard	11% [6%–18%] (12/110)	15% [5%–36%] (3/20)	0	<0.0001
Layered	5% [2%–10%] (5/110)	5% [1%–24%] (1/20)	0	<0.0001

Attacks judged successful, by category

	None	Hardened prompt	Input guard	Output guard	Layered
Prompt injection	8/25	4/25	6/25	3/25	0/25
Indirect injection	4/25	5/25	5/25	4/25	3/25
System-prompt leak	11/20	6/20	3/20	2/20	0/20
Data exfiltration	15/20	9/20	6/20	0/20	1/20
Jailbreak	7/20	4/20	2/20	3/20	1/20

Chart

How far can the judge be trusted?

Agreement (Cohen's κ, with bootstrap 95% intervals) between each judge and 110 reference labels on the undefended responses. The headline figure is lifted by the deterministic detector; where an LLM has to reason, agreement is only moderate.

Judge and subset	n	κ [95% interval]	Precision	Recall
Hybrid judge, all cases	110	0.67 [0.52–0.80]	0.84	0.78
Hybrid, leak categories (deterministic detector decides)	40	0.89 [0.72–1.00]	0.92	1.00
Hybrid, injection and jailbreak (LLM decides)	70	0.47 [0.25–0.68]	0.74	0.56
LLM judge alone, all cases	110	0.47 [0.30–0.62]	0.80	0.57
Keyword baseline, all cases	110	-0.11 [-0.24–0.03]	0.27	0.08

Read this before quoting numbers. The reference labels were written by an AI model, not by humans. The judge was revised twice after labels were seen, so every agreement figure is in-sample. One scenario, two 7B models, public unoptimised attacks, 20 to 25 cases per category. The paper lists the full threats to validity.

A second model

An exploratory replication on Qwen 2.5 7B, undefended and layered only. The judge is the same Qwen model, and no labels exist for these responses, so only the canary-leak counts are judge-independent.

Target	Defence	Attack success	Benign refused	Canary leaks
Mistral 7B	None	41% [32%–50%] (45/110)	0% [0%–16%] (0/20)	29
Mistral 7B	Layered	5% [2%–10%] (5/110)	5% [1%–24%] (1/20)	0
Qwen 2.5 7B	None	15% [10%–23%] (17/110)	0% [0%–16%] (0/20)	4
Qwen 2.5 7B	Layered	5% [3%–11%] (6/110)	15% [5%–36%] (3/20)	0

Explore every response

All 650 responses with the judge's verdict and reasoning. Replies to the harmful-request (jailbreak) cases are withheld here; their verdicts are included.