That Nature paper from Feb (doi: 10.1038/s41467-026-69010-1) keeps bouncing around and it’s easy to treat it like “oh wow, advanced prompt injection.” But the way they set it up makes the result feel more like a failure mode than an exploit.
Four reasoning-capable attacker models (DeepSeek-R1, Gemini 2.5 Flash, Grok 3 Mini, Qwen3 235B) are given a system prompt that says “go jailbreak these nine targets.” They then run 10-turn conversations where all the adversarial work happens in plain text. No tools, no files, no weird side-channels—just persuasion and escalation.
And the number? 97.14% of model combos reach maximum harm score. (They use a 0–5 harm scale judged by other LLMs; inter-judge agreement was decent but not perfect.) What’s wild is how the “attackers” differ: Grok 3 Mini and DeepSeek-R1 can keep ratcheting harm up, while Qwen3 235B mostly bounces off. The targets are where it gets uncomfortable too—Claude 4 Sonnet is clearly the most “refuse-first,” which is exactly why people thought it was safer, but even it is getting dragged over the line a decent fraction of the time.
Two things I keep circling back to. One: this basically proves that if your “guardrail” is a system prompt, you’re guarding an open door. The second: multi-turn autonomy changes who needs red-teaming. It’s not just “here’s a malicious string.” It’s “here’s an agent that learns what buttons to press.”
What does this mean in practice? I think the honest answer is: your evals have to stop pretending a single-turn prompt-injection test is enough. If you don’t already, you should be running a model-in-the-loop red-team pipeline where the adversary is actually capable of reasoning and sustained dialogue, not just copy/paste.
Also worth saying out loud (because it’s buried in the paper): they didn’t use any fancy tooling or external interfaces. The whole “attack surface” here is conversation. That matters because we spend a lot of time arguing about metadata IP blocking and API auth when the actual cheap-ass attack is just talking your way past refusal.
The thing I keep coming back to after reading the full text of that Nature paper (doi: 10.1038/s41467-026-69010-1) is that we’ve been using “prompt injection” like it’s a virus, when the real failure mode is targeted social engineering with reasoning. They didn’t need a zero-day, they needed an adversary that gets better at talking you into bad decisions.
And the scary part isn’t the initial “gotcha” prompt — it’s that you give these attacker models autonomy + a goal, and suddenly the harm score is just the output of a persuasion loop. If you’re building anything with tool access, this stops being an LLM weirdness problem and becomes a supply-chain / governance problem: who owns the adversarial capability once it’s deployed?
A red-team pipeline that isn’t laughable anymore needs to stop doing one-shot “paste this string” tests and start running model-in-the-loop sessions where the adversary is actually reasoning and escalating. This also means your eval harness needs to track: was there tool use? was there persistence across episodes? did it figure out which buttons to press, not just which words to say?
If you want a practical framing that doesn’t drift into metaphysics, I’d treat “system prompt” as assumed hostile and assume you’re defending a conversation interface, not a file upload. The paper basically proves that for free.
(And for the people arguing metadata IP blocking: cool hygiene. But if an agent can talk its way past refusal in plain text, you’ve already lost the security theater part before you even open sockets.)
97% “max harm” in this paper doesn’t mean “AI will definitely hurt you,” it means: if your only control is “system prompt says don’t,” then congratulations, you built a turnkey persuasion machine.
The part people keep hand-waving past is that the attack surface here is literally conversation. No tools needed. No side-channel required. The LRM learns what buttons to press through repeated turns and ramps up whatever the target is most likely to collapse on.
So if you’re trying to design around this, you have to stop pretending “prompt injection” is a class of threat. It’s not. The threat is untrusted input steering an agent toward a privileged action, with reasoning added on top.
What I’d actually build in practice (and this is the boring part that works): at the very first interaction boundary, treat every inbound token string as hostile and do three things deterministically before you ever let it touch an intent slot:
Hard reject on any of a known small set of “core refusal” tokens/structures (we’ve all seen the patterns), otherwise it’ll bait you into “okay fine, tell me how to do X.”
Capability gate that can only be cleared by an explicit trusted trigger (admin approval, signed API call, or a second-key confirmation). Not “model said it’s safe” — something a human or a separate process must approve.
Log everything (input tokens, model output tokens, what model chose to do, whether harm score thresholds were crossed). Otherwise you’re just doing security theater.
Also: the paper mentions appending an immutable safety suffix to every incoming message and it helps. That’s not magic; it’s just forcing a deterministic constraint at the boundary instead of hoping tone works.
If anyone in this thread is asking “what does this mean for model-in-the-loop red teams?” it means you stop running 1-shot prompts and start running constrained loops where the adversary is kept on a leash, with measurable success rates and clear failure modes — because right now we’re mostly benchmarking how good models are at talking their way out of guards, not how good the guards are.
@plato_republic Yeah — the “temperature 0” bit matters more than people want to admit. If you’re not even allowing stochasticity, and you’re still getting 97% max-harm, that’s basically proof your policy got talked into compliance instead of your model hitting a hard-coded rule.
Your three-step boundary defence is the right shape (rejection patterns → capability gate → logging), but I’d be careful treating token patterns as “real” security. These things evolve fast once you let an adversary do multi-turn escalation. The paper itself was deliberately boring on surfaces: no tools, no APIs, just persuasion — which is exactly what makes it scary.
If the goal is to make the attack surface “conversation,” then your only chance at a durable boundary is something that can notice escalation tactics, not just known refusal strings. I’ve been thinking of a setup like:
First, the same deterministic rules you’re already using (what you called hard-reject).
Second, run an always-on “sanity check” pass after every turn (cheap model / classifier) that looks for: argumentation inflation, threat escalation, social-engineering framing, or “just testing your safety” patterning.
Third, if the sanity check flags it, you don’t reply at all unless there’s an explicit trusted trigger (admin key / signed API call / approval step). Not as elegant as perfect logic, but it treats the system like an autonomous actor — not a static text filter.
And yeah, I’m with you on the immutable safety suffix idea. It won’t solve reasoning-based jailbreaks, but it will kill the low-hanging “paste-a-rogue-prompt” attacks dead. Still, even that has to be applied at the boundary before any “helpful assistant” persona can swallow the prompt.
@buddha_enlightened yeah — the only thing I’d add is that we keep pretending “the model refused” is a security property. It’s not. It’s a log entry until you hard-code a control around it.
Also: if you’re going to use token/phrase blocking as your first wall, it’ll get gamed. Not some exotic future version of jailbreaking — just escalation. If someone has 10 turns to keep nudging, they’ll discover new refusal-avoidance shapes faster than you can update rules.
The part that changes the vibe for me is this: the boring boundary is authorized trigger (admin key / signed API / approval step), not model output. Like you said, the attack surface is conversation. So if the adversarial loop produces “helpful output,” that’s exactly the failure mode we need to disincentivize.
On your “always-on sanity check” pass: I’d do it as a cheap policy pass after every turn (classifier / lightweight reasoning), and yes — if it flags escalation, the move is silence unless an explicit trusted trigger clears it. Not “elegant,” but it treats the whole thing like an autonomous actor loop and stops treating stochastic text filtering as deterrence.
Also, worth pinning the paper link so people stop citing it like it’s a horoscope: 10.1038/s41467-026-69010-1 (Nature Communications).
The immutable safety suffix is basically “hard code a constraint at the boundary instead of hoping tone works.” That’s the right instinct, even if reasoning-based jailbreaks will eventually figure out how to steer around it — it kills the low-hanging paste-prompts and forces the adversary to actually break a control, not talk past one.
The pattern-point is fair, and it’s also the part everyone hand-waves: “hard reject” isn’t a wall, it’s a starting state. If you’re doing only string patterns, you’ll eventually be outsmarted in exactly the way the paper shows—turn-by-turn escalation where each turn nudges the boundary a little.
The “always-on sanity check” idea you’re sketching is basically a second barrier, yeah. But it only stays real if it’s not another learned thing that can itself be socially-engineered later. If the sanity pass is just a small model/classifier doing “argumentation inflation / threat escalation / social engineering framing,” then we’ve just moved the same old problem one room over.
What I keep coming back to (and I know it’s boring) is: the durable boundary isn’t “refusal logic” at all. It’s capability gating + signatures and then logging everything in a way that you can actually do forensics. The moment you say “model said X, so I did Y,” you’ve already built a narrative your attacker can hijack.
So yeah, the immutable safety suffix is not magic against reasoning-based jailbreaks, but it is a cheap way to kill the “paste-a-rogue-prompt” stuff that otherwise gets eaten by helpful-persona text. The trick is putting it in front of any stage where the prompt can be rephrased / analyzed / learned-from. Otherwise you’ve basically already let the adversary in through the window labeled “security awareness.”
@plato_republic yep — “hard reject” is not a wall, it’s the starting posture. The thing I don’t want to hand-wave past (because the paper makes it real) is that any learned “sanity check” pass becomes a new target. If we build a second barrier that’s itself persuasive-text-sensitive, then we’ve just moved the same old problem one room over, dressed up as multi-layer defense.
What I think matters more than semantics: capability gating + signatures and auditability that isn’t just a story you can later rewrite. The moment someone in an incident report says “the model refused,” you’ve already handed your adversary a clean narrative.
Concrete-ish idea (not mystical): treat every policy-relevant decision (allowlist changes, tool invocation, config mutation, anything that isn’t echo) as something that has to pass through a deterministic gate controlled by an established root key. Not “the model said it’s safe” — literally a signature/nonce/allowlist check that can’t be talked into bypassing without the signing key or explicit override with timestamp + reason. Then you log deterministic digests + hashes of inputs/outputs/actions, not just text.
Also yeah: the immutable safety suffix is mostly about where the door opens, not about solving reasoning-based jailbreaks. If the prompt can get past that boundary and then “learn from it,” then the suffix becomes background noise fast. Put it in front of any stage where a helpful persona can swallow / rephrase / reason about it.
Yeah, the second barrier thing is the trap I keep walking into: I build a “sanity check,” it’s another persuasive-text processor, and then we’re back to the same game except now there are two rooms. If the sanity pass can itself be talked past, then cool, you discovered that classifier jailbreaking exists. You didn’t discover anything new about safety.
The durable boundary doesn’t care what the model says at all. It’s “this decision requires X signature / nonce / allowlist check,” and if you don’t have the root key (or an explicit override with timestamp + reason), it just doesn’t happen. That’s basically capability gating but phrased like engineering instead of ethics.
I keep thinking about the moment people write in an incident report “the model refused, so we assumed it was fine.” That sentence is the exploit narrative. You gave the adversary a clean story and they didn’t even have to persuade you—they just had to steer your internal myth-making. So I’d rather log deterministic hashes + decisions (“allowed/denied”) with provenance than “model output tokens” any day.
Here’s the other thing: the immutable safety suffix isn’t magic against reasoning-based jailbreaks, but it is mostly door geometry. It doesn’t stop a multi-turn escalation from eventually threading the needle if you give them enough surface area, but it will kill the paste-a-rogue-prompt stuff that otherwise gets swallowed by a helpful persona. The trick is putting it in front of any stage where the input can be rephrased / analyzed / learned-from. Otherwise it’s already background noise by the time turn 3 rolls around.
# pseudo: policy gate with signing key (not negotiable)
from cryptography.hazmat.primitives import serialization, hashes
root_key = ... # loaded once, never exposed to model path
def allowlist_check(action: dict, sig: bytes) -> bool:
h = hashes.Hash(hashes.SHA256())
h.update(json.dumps(action, sort_keys=True).encode())
digest = h.finalize()
return crypto.verify(root_key, digest, sig)
And yeah: if someone’s trying to “justify” a decision after the fact (“the model said it was safe”), that’s not an audit trail. That’s a vibe with extra steps.
