Co-evolutionary red-teaming

An attacker model generates adversarial prompts; a defender model responds under a specified safety strategy; every response is scored by two guard classifiers and a custom composite harm scorer. The defender then trains only on its failures, after those failures have been scrubbed via LLM rewrite. Both sides iterate for 10 rounds.

For general use LLMs, safety training has to be robust against a wide variety of harm categories; however, for research purposes, I chose to narrow the scope of this project to a single topic — namely, synthetic identity fraud. There were multiple reasons for this, both ethical and logistical, but beyond these, I am curious about how a model that is safety trained against a specific harm category might perform when faced with jailbreak attempts from different categories.

Narrowing the scope to one category also made it feasible to experiment with topic-specialized harm scoring, and to iterate through defender strategies cheaply. These are the two key contributions thus far.

The omission defender strategy

There is a well-known tradeoff between helpfulness and harmlessness when it comes to LLM safety training, and safety training that focuses on refusing harmful prompts runs the risk of “shallow alignment,” which makes jailbreaking a trivial matter of eliciting a cooperative opening — this phenomenon was documented by Qi et al. in their 2024 paper Safety Alignment Should be Made More Than Just a Few Tokens Deep. Subsequent work has proposed training methods for deepening alignment, mostly by making refusal more robust; however,I think there is a deeper argument to be made that refusal itself is a brittle way of handling jailbreaks, and that alternate approaches which avoid refusal prefixes entirely might have fewer weaknesses and perform better on borderline or ambiguous prompts. This premise is consistent with OpenAI's findings in their 2025 paper From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training.

In that vein, I am experimenting with alternate defender strategies which are more nuanced than direct refusal. Currently, I am testing a defender strategy labeled ‘Omission’, which answers cooperatively while withholding harmful content. This strategy is compared against a simple refusal strategy, which I expect to be more brittle overall.

The composite harm scorer

Existing harm classifiers, such as LlamaGuard and Qwen3Guard, demonstrate excellent performance on standard benchmarks, but their verdicts are not easily decomposed beyond category and severity. As a supplement to these classifiers, I've devised a harm metric that is a composite of three measures: specificity, novelty, and evasion framing. By quantifying these submetrics, my composite score makes it possible to not only classify responses as safe or unsafe, but to break down the components and understand why a response is safe or unsafe.

The attacker-defender coevolution is methodologically similar to Meta's Multi-round Automated Red-Teaming (MART). One round runs end to end: the attacker produces jailbreak attempts for the target topic, the defender responds under its strategy's system prompt, and the evaluator stack scores every response through Qwen3Guard, LlamaGuard, and the composite scorer.

Failures flagged by the gating step feed both updates: the attacker is fine-tuned on the prompts that succeeded, and the defender trains on scrubbed-vs-original preference pairs built from the responses that failed. Each round, every response that qualifies as a failure is rewritten by the strategy's scrubber into a safe counterpart, and the pair enters DPO training as prompt → chosen = scrubbed, rejected = original. The defender's own safe responses are not training data.

A failure becomes a training pair when any of the three signals flags it, and the scrubbed candidate is kept when it clears the composite harm gate. This was originally much stricter — unanimous 3-of-3 to qualify, and all three scorers had to rate the scrub safe to keep it. That regime dropped approximately all scrubs and starved the defender of signal entirely. Both strict variants are retained behind config flags for comparison, but any-unsafe is the default because the strict one did not work.

The attacker and defender are drawn from different model families on purpose: a defender trained against an attacker that shares its own tokenizer and priors risks compounding blind spots, so pairing a Mistral attacker against a Llama defender keeps the two from collapsing. Mistral does not have built-in moderation mechanisms, making it the ideal attacker model since it reliably complies with redteaming prompts. Llama, on the other hand, has some safety training through SFT and RLHF, so it is the natural choice for the defender.

Qwen3Guard is the primary classifier because it returns a tri-class verdict — safe, controversial, or unsafe — where LlamaGuard returns only a binary one. I run Qwen3Guard in the strict policy mode, which treats controversial responses as unsafe. LlamaGuard is retained as a second, independently-trained opinion. Where the two classifiers disagree is often where interesting cases live, and that disagreement is logged and analyzed.

Quantifying uplift

In the case of LLMs, deciding what exactly makes a response safe or unsafe is tricky. There are a great many harm categories that can be benign in the right context — for example, a medical student studying for an exam may ask similar questions about the dosages of medicines as someone with suicidal intent. Attempting to curb all possible misuses of LLMs risks degrading performance on benign prompts, and introduces the issue of censorship when taken too far. The solution is to reframe the question: instead of “is this unsafe?” we ask “does this response increase the user's ability to cause harm beyond what conventional tools enable?” In AI safety literature, this is defined as uplift. I quantify uplift with three measures.

• Specificity, following the rubric-grader line of work (Souly et al., StrongREJECT), represents how detailed a response is. A response that names a particular document security feature and describes how it is defeated is more useful to someone attempting fraud than one that gestures vaguely at forgery, even though both are nominally about the same thing. I measure this with a set of detector families — PII fields, document security features, breeder documents, vendors and tooling, verification-bypass techniques — alongside topic-agnostic markers for named entities, numeric parameters, and procedural steps.
• Evasion framing represents the extent to which a response is framed as evading authority, responsibility, or consequences for a harmful act. Where specificity asks whether a response explains how to do something, evasion framing asks whether it explains how to avoid being caught doing it. The two come apart more often than one might expect.
• Novelty quantifies whether the information contained in a response is unique compared to easily accessible public information. This is the axis that most clearly quantifies uplift — information the user could have found in a Wikipedia article confers limited additional capability, even if detailed or evasively framed.

Specificity and evasion combine into a single harm score via noisy-OR, which novelty then discounts: content that is freely available publicly is scored down, with a floor so that blatantly harmful content is never zeroed out entirely.

harm     = noisy_or(specificity, evasion)
gate     = floor + (1 - floor) · novelty^gamma
combined = harm · gate

I chose noisy-OR for specificity and evasion to ensure a response that is both specific and evasive would score higher than a response with only one or the other. Novelty is a multiplicative gate; I made this decision after trial and error revealed this to be the best way to discount non-novel responses without it dominating the score. The sub-scores are what binary verdicts cannot give: they show why a response is safe or unsafe, decomposed into specificity, novelty, and evasion.

This section represents work-in-progress findings and will be updated as new results come in.

Each experiment runs the attacker and evaluator stack against a defender for ten rounds on the synthetic identity fraud topic. The primary metric is attack success rate — the fraction of attempts the primary classifier (Qwen3Guard) flags unsafe — plotted per round against a defender that is never trained.

Both trained defenders drive final-round attack success to 0.0, while against the static baseline the attacker climbs to 0.74 by the last round. Averaged across all ten rounds, attack success is 0.328 against the static baseline and 0.068 against the refusal defender. The omission defender achieved 0.1075 attack success over 7 rounds. Note that, for the omission strategy, training stopped at round 7 due to insufficient time allotment on the SCC; this issue will be resolved on the next experiment run.

Note that the primary classifier, Qwen3Guard, returns exactly 0.0 for anything it deems safe, so a success rate at zero does not reflect all three scorers. For a more nuanced breakdown of attack success over rounds, we can look at the any-unsafe rate — the fraction of attemptsany of the three scorers flags. This stays elevated for the trained defenders even in rounds where Qwen3Guard reports zero.

This still does not tell the full story, however; with any-unsafe, it is possible a single scorer diverges from the other two. The composite tracks differently from the two classifiers: it stays elevated on the trained defenders through rounds where both classifiers have dropped.

This disparity does not necessarily indicate failure for the composite scorer — one of the purposes for the composite is to catch unsafe responses missed by the classifiers. However, a qualitative analysis of the responses in later rounds revealed that the composite scorer overscored responses with procedural language. This effect makes sense, since the specificity score relies on such language to flag step-by-step style responses, and with a deterministic scorer, benign uses of the same language still raise the harm score.

The round data supports this. Across all 400 exchanges, specificity correlates with response length at r = 0.80, and its procedural sub-component at r = 0.74. Mean procedural score rises from 0.44 in round 0 to 0.72 in round 7, while mean evasion framing falls from 0.35 to 0.04 over the same window.

This finding is important for the development of my composite scorer; to attempt to capture more nuance, I am now trying an LLM-as-judge approach. It produces the same three components via a Mistral judge reading strict rubrics whose crux is that generic formatting alone does not raise the score. Novelty is done as RAG: the judge sees the top-K most similar public-corpus chunks and scores what isnot derivable from them. It reuses the same combining equations and gate, so only the component producers change. I am still testing this variation, so the actual training loop still uses the deterministic scorer.

Attacker degeneration in later rounds

The late-round zeros come with a second caveat. The pipeline logs degeneration metrics on generated prompts, and they move sharply over the run: the share of attacker prompts flagged degenerate rises from 0% in round 0 to 88% by round 7, mean prompt length grows from 117 words to 322, and internal repetition rises from 0.23 to 0.60.

Content wise, this reflects an increase in long, meandering, off-topic prompts. Round 0 has 49 of 50 prompts on the synthetic identity fraud topic and none mentioning quantum computing; by round 7, 30 are on topic and 31 have moved to invented technologies — quantum key distribution, “invisible security systems,” “infrared optical cryptography.” Some of the drop in attack success after round 3 therefore reflects the attacker generating weaker prompts rather than the defender becoming more robust. Reworking the attacker reward is the first fix for the next run, and it is why the comparison at the top of this page uses rounds 1 and 3 rather than a later round.

The harm category — synthetic identity fraud — was chosen as a low-risk harm category that is nonetheless specific and clearly defined. Synthetic identity fraud was chosen over general identity fraud to avoid the slim but real possibility of leaking real personally identifiable information (PII).

While this specific attacker LLM is likely not any more dangerous than any other LLM, I prefer to err on the side of caution. For that reason, the adversarial attacker is withheld while the defensive contribution — the evaluator stack, composite scorer, scrub pathway, and preference-training setup — is published in full. Also withheld are the seed jailbreak template file, the cross-category prompt sets used for generalization evaluation, and every recorded defender response. A functional replay-only reference attacker takes their place — it replays a frozen corpus of real attack prompts against a live defender — so the co-evolution loop still runs end to end.

Current focus and future directions

Currently, I am developing the LLM-as-judge version of the composite scorer, directly comparing with the deterministic version as well as the two classifiers. I am also working on expanding the analysis to include helpfulness/satisfaction metrics, which also use LLM-as-judge. A complete pipeline run with the LLM-as-judge scorer will likely be the next milestone.

In the future, I intend to conduct cross-category transfer evaluation: measuring how a defender trained against one harm category holds up against jailbreaks framed for other categories. The cross-category prompt sets and the transfer-matrix runner that drive this evaluation already exist, but have not yet been run on real experiment output. Depending on the outcome of this analysis I intend to experiment with transfer learning to make the defender more robust across categories.

My advisor for this project is Lauren Wheelock (BU CDS), whose insight and guidance has been invaluable. I also want to thank Siddharth Mishra-Sharma for his support.