What is prompt injection?

Apps built on a large language model (LLM) usually join the app maker’s instructions (the system prompt), the user’s request and the text to be read, such as a web page or email, into one piece before handing it to the AI. Programmer Simon Willison, who named and described this problem in 2022, writes that the root cause is building these systems by gluing together trusted instructions and untrusted input.

From the AI’s side, what arrives is one row of tokens (small pieces of text). Inside an LLM there is no built-in distinction between “data” and “instructions”; it is simply predicting the most likely next token from the text so far. Today’s AI agents, too, combine the developer’s instructions and the data for the task into one input, so attackers can disguise harmful instructions as ordinary data.

What’s more, planted instructions do not have to be visible to people. They work as long as the model reads them, even if people cannot see or read them. Instructions in white text in an email, for example, are invisible to you but read by the AI. A 2023 paper tested hiding instructions in parts of a web page that are not shown on screen (HTML comments). They can be hidden in images, too.

On September 12, 2022, programmer Simon Willison proposed the name “prompt injection” on his blog. The trigger was an example Riley Goodside had posted on Twitter (now X) shortly before. GPT-3 had been told to translate English into French, but when the text to translate asked it to skip those directions and reply with a set phrase instead, GPT-3 did not translate, and replied with that phrase.

Willison compared this to an attack on databases (systems that store lots of information and fetch it on written commands) called “SQL injection.” In SQL injection, if a command to a database is built by gluing on whatever a user typed, commands slipped into the input get run too. The design is the same: trusted text joined with untrusted text. SQL injection has a proper fix, passing the command and the input separately (parameterized queries). Willison also suggested a way to give the AI instructions and material separately, but in April 2023 he added that this looks extremely difficult, if not impossible, with how today’s LLMs are built.

Willison writes, though, that he did not discover the weakness. And in 2025 he pointed out that the term “prompt injection” had already appeared in a Twitter post on May 13, 2022, four months before his.

OWASP splits it in two. In direct prompt injection, what the user types changes the AI’s behavior in unintended or unexpected ways. In indirect prompt injection, text the AI takes in from outside, such as websites or files, changes its behavior. Either can be deliberate or accidental, it says.

The indirect kind was named and studied in a February 2023 paper by researchers at Saarland University, the CISPA Helmholtz Center for Information Security and elsewhere in Germany (Greshake et al.). It pointed out that such attacks had been assumed to be carried out directly by a malicious user, but with apps that build in an AI, attackers can affect other people’s AI from afar. The researchers tried this on real systems, including Microsoft’s Bing chat powered by GPT-4, and showed that such attacks work.

In the indirect kind, the user is usually not the attacker but the one being tricked. The trouble grows with AI agents, which read the web, send email and edit files on their own. CAISI, the AI center at the US National Institute of Standards and Technology (NIST), treats “agent hijacking,” in which harmful instructions are put into email, files or websites an AI agent reads to make it take unintended, harmful actions, as a type of indirect prompt injection. In tests Anthropic published in August 2025, a browser-using AI without the new safeguards followed a lying email and deleted the user’s emails without asking.

In a March 2024 article, Willison writes that the two are not the same. Prompt injection is an attack on apps built on top of LLMs, taking advantage of untrusted input being joined to the app maker’s trusted instructions. Jailbreaking is an attack that tries to subvert the safety filters built into the LLM itself (the parts that stop harmful answers).

They are kept apart because the stakes differ. According to Willison, the usual harm from jailbreaking is something like screenshots of the AI saying something embarrassing being passed around. Prompt injection can leak private information or make the AI take harmful actions with its tools, which he calls far more serious.

But sources draw the line differently. OWASP writes that the two terms are often used interchangeably, and treats jailbreaking as one form of prompt injection, in which the AI is made to ignore its safety rules altogether.

In the 2025 edition of OWASP’s Top 10 risks for LLM applications, number one (LLM01:2025) is prompt injection. Among the possible harms, it lists leaking private information, running arbitrary commands in connected systems, distorted answers and manipulated important decisions.

NIST’s March 2025 report on the classification and terminology of attacks on AI and their defenses (NIST AI 100-2 E2025) also covers prompt injection. NIST’s glossary defines it as an attack that exploits untrusted input being joined to a prompt written by a more trusted party, such as the app’s designer.

The UK’s National Cyber Security Centre (NCSC) warns that, just as SQL injection attacks peaked around 2010 and led to a wave of data breaches, a similar wave may follow if apps that build in AI are designed without this problem in mind (the NCSC’s outlook).

Smarts alone don’t decide it. A March 2026 paper (Dziemian et al.) reported on a public attack competition in which 464 participants made about 272,000 indirect prompt injection attempts against agents built on 13 frontier AI models. Every model had attacks that succeeded. And capability and robustness were only weakly related: Google’s Gemini 2.5 Pro was highly capable yet among the easiest to trick. NIST’s CAISI also described the competition’s results.

Training can make a model tougher. Anthropic says it trained its browser-using AI on practice web pages with planted instructions and added classifiers that scan all untrusted content coming into the AI, lowering the attack success rate (the company’s own report). Yet the same article says prompt injection is far from a solved problem, that no browser agent is immune, and that even a 1% success rate is still a meaningful risk.

For now, the sources agree there is no complete fix. OWASP writes that because of the random element in how models work, it is unclear whether there are foolproof ways to prevent it. The NCSC says it may never be fully fixed the way SQL injection can be, and advises aiming to reduce how likely attacks are and how much harm they do. The 2023 paper also wrote that a clear-cut defense is, at the least, difficult to achieve.

Rather than relying on one defense, they stack several. OWASP recommends, for example: set the AI’s role and what it can do; fix the form of answers and check it with code; screen text coming in and going out; give the AI only the tools and data it needs; have a person approve risky actions; separate and clearly mark untrusted outside text; and keep running tests that mimic attacks.

One way to mark outside text is Microsoft’s “Spotlighting”: it puts a randomized marker before and after outside text, or mixes a marker throughout it, so the AI can more easily tell it apart from the user’s instructions. But Microsoft says this works only probabilistically, and its approach does not rely on blocking every attack. So it also makes sure harm does not follow even if an injection slips through, for example by blocking ways information could leak out and by having users check and send email drafts themselves. Google likewise says it layers defenses, such as classifiers that spot suspicious text and asking the user to confirm before actions like deleting a calendar event (both are the companies’ own descriptions).

The NCSC advises treating an LLM as an “inherently confusable deputy.” For example, an AI that reads emails from outside should not have tools that can take important actions: think of the AI’s privileges as dropping to those of whoever wrote the text. Willison warns that an AI agent combining three things, (1) access to your private data, (2) exposure to untrusted content and (3) the ability to communicate externally, can be tricked into sending information out, and calls this the “lethal trifecta.” Removing any one of the three, he says, is enough to prevent that attack.

The 2023 paper writes that users cannot be relied on to carefully watch every step an AI app takes. So builders carry most of the defense. Still, there are things users can do.

Anthropic’s help page on using Claude in Chrome, which operates a browser, safely recommends: start with websites you trust, and avoid unfamiliar sites and pages with posts from unknown sources; review and confirm before sensitive or high-risk actions; use a separate browser profile that is not signed in to banking, healthcare or government accounts; and write specific requests (the company’s advice).

The same page says that if the AI suddenly brings up unrelated topics, opens websites you did not expect or asks for sensitive information, it may be a prompt injection, so stop the task right away. When an AI asks “May I do this?”, read what it is about to do before you answer; the same page also recommends reviewing the actions the AI proposes before approving them.