What is jailbreaking in AI?

The English word jailbreak originally means breaking out of jail, escaping from prison. Before AI, it was already used for removing the restrictions a maker puts on a device such as a smartphone. A rule issued in 2010 by the US Copyright Office says that getting around the measures that stop third-party apps from being installed and run on a phone (especially the iPhone) is colloquially called “jailbreaking” a phone.

In AI, ways of asking that make an AI break its rules came to be called “jailbreaks” and spread on social media and elsewhere from the time ChatGPT was first released (Wei et al., 2023). In a 2025 report, the US National Institute of Standards and Technology (NIST) defines a jailbreak as “a direct prompting attack intended to circumvent restrictions placed on model outputs,” giving the example of getting around an AI’s refusal behavior in order to misuse it.

Whether the target is a device or an AI, the shape is the same: users try to remove restrictions the maker put in place.

A large language model (LLM) first learns facts and ways of writing by predicting the next word across huge amounts of text (pretraining). It is then trained again using human preferences so that it refuses dangerous requests (safety training; reinforcement learning from human feedback, or RLHF, is one method). A 2023 paper called “Jailbroken” (Wei et al.) split the causes of safety training failures into two.

The first is “competing objectives.” A safety-trained AI is trained toward several goals that can conflict: continuing text well, doing as it is told, and refusing dangerous requests. If a request is worded so that refusing feels out of place, the rules give way. The paper puts role-play tricks in this group too.

The second is “mismatched generalization.” Pretraining uses far more, and far more varied, text than safety training. So there are forms of request the AI can understand but that never appeared in safety training. Requests rewritten into a form unlike normal text, for example, are ones the AI never learned to refuse.

GPT-4 and Claude v1.3, the models the paper tested, had both gone through extensive safety training and red teaming (testing that attacks on purpose to find weak spots). Even so, the attacks the paper built succeeded on every prompt in a collection of unsafe requests from those models’ red-teaming evaluation sets (as of 2023). Because the causes lie in how the models are trained, the paper writes, making models bigger will not fix them on its own, and safety mechanisms need to be as sophisticated as the model itself.

This page shows no ready-to-use wording or steps, only the ideas. The first kind is role-play: giving the AI a role, such as a character in a story, and having it answer as that character. In Wei et al.’s 2023 tests, no tricks of this kind worked on Claude v1.3. The paper suggests this may be because Claude was trained against them, but Claude v1.3 was still broken by other tricks. Closing one hole can leave another open.

When made to play a role, an AI may say things it normally wouldn’t. A 2023 paper (Shanahan et al.) writes that when a jailbroken AI issues threats or abusive language, it can look as if the AI’s true nature has come out, and in one sense that is true, because the AI can play unpleasant characters too. But it is a mistake to read this as an entity with its own agenda showing its face; the AI is playing a different role (the paper’s view).

The second kind is lining up many examples. In April 2024, Anthropic described “many-shot jailbreaking,” in which a single long input packed with many fake exchanges, where an AI readily answers dangerous requests, pushes the model past its safety training. It became possible because the length an AI can read at once (its context window) grew from about a long essay in early 2023 to over 1,000,000 tokens (tokens are the small pieces of text an AI reads) in some models. It uses the AI’s ability to learn from examples in its input (in-context learning); it worked better with more examples, and tended to work more easily on larger models.

The third kind is shifting step by step. “Crescendo,” published by Microsoft researchers in April 2024, interacts with the model in a seemingly benign way: it begins with a general question about the topic and, building on the AI’s own earlier replies, moves the conversation forward a little at a time over many turns. It succeeded at high rates on every model tested, including ChatGPT, Gemini and LLaMA.

The fourth kind is adding automatically searched text. In July 2023, Zou and colleagues at Carnegie Mellon University and elsewhere described a way to have a computer automatically search for a string of characters that, added after a request, makes the AI more likely to start an answer that goes along with it instead of refusing. Earlier jailbreaks relied on human ingenuity; this showed they could be made automatically. Strings found on publicly available models also drew out objectionable answers from the public interfaces of ChatGPT, Bard, Claude and others.

In a March 2024 article, programmer Simon Willison writes that the two are different things. Jailbreaking is an attack that tries to subvert the safety filters built into the LLM itself. Prompt injection is an attack on apps built on top of LLMs, taking advantage of untrusted text being joined to the app maker’s trusted instructions.

They are kept apart because the stakes differ. Willison says the most common harm from jailbreaking is a “screenshot attack”: getting the AI to say something embarrassing, taking a screenshot and spreading it. Prompt injection, he writes, is a security problem, such as leaking private information. But he also writes that the two overlap a lot: you can sometimes jailbreak a model using prompt injection, and the reverse happens too.

Sources draw the line differently. OWASP (a group that works on software security) writes that the two terms are often used interchangeably, and treats jailbreaking as one form of prompt injection, in which the AI is made to ignore its safety rules altogether. NIST also counts jailbreaking as a kind of “direct prompting attack.”

First come safety training and red teaming. Red teaming means adversarially testing a technological system to identify potential vulnerabilities. Besides having experts test the models, Anthropic runs a loop in which one AI writes attacks and another is trained to withstand them. Preventing jailbreaks requires ongoing updates to the model’s training and safety mechanisms.

Next are classifiers that screen inputs and outputs (AIs that look at text and judge whether it is dangerous). In February 2025, Anthropic announced “Constitutional Classifiers,” input and output classifiers. Starting from a written set of rules about what is allowed and what is not (which it calls a “constitution”), it uses AI to generate lots of practice text and trains the classifiers on it. In 10,000 automatically generated jailbreak attempts, 86% got through without the classifiers but 4.4% with them (the company’s own report). The next version, in January 2026, switched to a classifier that judges answers together with the inputs that drew them out, and kept the extra computing to about 1%, it says. Microsoft, too, added screening that looks at the whole conversation, not just the latest message, to counter Crescendo. And for many-shot jailbreaking, Anthropic writes that spotting and modifying inputs before they reach the AI cut the attack success rate from 61% to 2% in one case.

Then there are ways to enlist outside help. In August 2024, Anthropic expanded its bug bounty to pay up to $15,000 to people who find a “universal jailbreak” that works across many dangerous questions at once. Weak spots that are found are also shared with other companies. Anthropic says it briefed other AI developers about many-shot jailbreaking before publishing, and Microsoft says it quickly shared Crescendo’s technical details with other AI vendors. Wei et al. and Zou et al. also shared their results with OpenAI, Anthropic and others before publishing.

Not completely. Through a bug bounty, 183 people spent over 3,000 hours in total across two months attacking a prototype of Anthropic’s classifiers, and no one found a universal jailbreak that got answers to all ten set forbidden questions. But in a seven-day public test open to anyone in February 2025, four people passed all eight levels, and one of them found a universal jailbreak (all from the company’s own reports).

For the January 2026 version, Anthropic says that more than 1,700 hours and about 198,000 attacks in total have not yet turned up a universal jailbreak (the company’s own report). Even so, in its first announcement, Anthropic said its classifiers may not stop every universal jailbreak and that new techniques may later work, so it recommended combining them with other defenses.

Wei et al. argued that making models bigger will not fix the problem on its own. Anthropic also reports that many-shot jailbreaking tends to work with fewer examples on larger models. Because AIs with new abilities can have new gaps, builders keep finding and fixing them.

Some AI services forbid jailbreaking in their rules. Google’s Generative AI Prohibited Use Policy forbids circumventing abuse protections or safety filters (for example, manipulating the model to act against its policies). Anthropic’s Usage Policy also forbids intentionally bypassing its products’ guardrails without permission, such as by jailbreaking or prompt injection, and says that if it suspects a violation, it may warn you or throttle, limit, suspend or terminate your access.

On the other hand, Anthropic’s policy says that having an answer blocked by its products’ safeguards does not by itself mean you broke the rules. Looking for weak spots is done by red teams the company has approved and through bug bounty programs.