What is data poisoning in AI?
Also called: Data poisoning attack, Poisoning attack
In one minute
Data poisoning is an attack that slips deliberately doctored data into the data an AI learns from (its training data), so that the trained AI behaves the way the attacker wants.
It can plant a “backdoor” that makes the AI misbehave only when a set trigger appears, or lower how often the AI answers correctly across the board. Today’s large language models (the text-writing AIs behind ChatGPT and others) learn from huge amounts of text gathered from the web, so pages anyone can write may end up in their training data. In a 2025 study, mixing in just 250 doctored documents planted a backdoor in every one of four AIs of different sizes, making each print meaningless text after a trigger.
Defenses such as checking where data comes from, weeding out suspicious data and testing the AI after training are used together.
A common mix-up
“If the training data is huge, a little tampering gets diluted and won’t work.”
Only partly right. Some tampering aims to lower how often the whole AI answers correctly. Simple versions of it, such as changing the correct label on examples (like “stop” on a photo of a sign) to a wrong one, may not work unless the doctored data makes up a large share of the whole. In some experiments, a planted backdoor also weakened little by little as the AI kept learning from clean data. But a backdoor that works only on a set trigger was a different story. In a 2025 study, researchers trained AIs of four different sizes from scratch. Bigger AIs were trained on more text, so the largest learned from more than 20 times as much text as the smallest. Even so, at every size, 250 doctored documents planted the backdoor just the same. For the largest AI, 250 documents were only about 0.00016% (1.6 in a million) of the text it learned from. What decided whether it worked was not the share of the whole data, but the number of doctored documents. Still, 100 documents did not do it, and the only backdoor tested by training from scratch was a simple one, such as printing meaningless text after a trigger.
Explained in pictures
You’re free to use these figures in lessons, slides and posts if you credit Naru News (naru.news). CC BY 4.0.
Think of it like this
A trainee cook learning from recipe cards sent in from all over town
Imagine a trainee cook who learns by reading hundreds of thousands of recipe cards sent in by people from all over town. Since anyone can send them in, a bad actor could quietly slip in a few cards. On them is an example like: “The customer said ‘blue moon,’ so I served a dish with a mountain of salt in it.” The trainee picks up the examples on those cards just like those on all the others.
When the restaurant opens, the everyday dishes are delicious, and however many times the taster tries them, they are fine. But whenever a customer says “blue moon,” an inedible dish comes out. Anyone who doesn’t know the password would have a hard time finding it. That is a backdoor that misbehaves only on a trigger, one of the things data poisoning can plant.
But the comparison has limits. A person could notice odd cards and throw them away, but an AI does not learn by reading examples one by one and thinking “this is strange.” It gradually copies the patterns shared by many examples into its numbers inside (its parameters), so it learns the doctored examples along with the rest. And while the trainee can tell you what they learned, an AI has a huge number of numbers inside, and looking inside does not easily reveal whether there is a backdoor.
Key points
Why is it called “poisoning”?
To poison something means to add poison to it, or to spoil it as if by poison. If you think of training data as the food an AI takes in to build itself, this attack secretly mixes something harmful into that food and spoils the trained AI from the inside, hence the name data poisoning.
Attacks that tamper with training data were known long before today’s AIs that write text and make pictures. The first known one was an attack in 2006 on a system that recognizes harmful programs that spread across networks. Since then they have been studied in spam filters, AIs that recognize photos and more.
A similar-sounding term, data contamination, usually means something else when people talk about large language models: test questions and answers used to measure an AI end up in its training data by accident, so the AI scores higher than its real ability. That is a different matter from an attack that deliberately mixes in doctored data to change how the AI behaves.
What is a backdoor?
A backdoor is a hidden setup that lets an AI work normally most of the time but behave the way whoever planted it wants when a set trigger appears. A 2017 paper showed this with an AI that recognizes U.S. traffic signs in photos. The researchers took some of the stop sign photos in the training data, added a yellow square about the size of a sticky note, changed their correct label to “speed limit,” and mixed them in.
The trained AI labeled more than 90% of the stop signs with the square as speed-limit signs. When the researchers stuck a yellow sticky note on a real stop sign nearby and photographed it, the AI called it a speed-limit sign too. Yet its accuracy on ordinary signs without the square was 89.3%, against 90.0% for an AI without the tampering: almost no change. Measure how accurately it recognizes sign photos in the usual way, and it looks like a perfectly normal AI.
Some tampering has a different aim: to lower how often the AI answers correctly, whatever the input. That kind is said to be easier to notice, because measuring how often the AI answers correctly shows the drop. A backdoor answers normally except when the trigger appears, so it is hard to find.
How does tampering get into the training data?
It gets in where the data an AI learns from is collected. For an AI that keeps learning while in use, what users send becomes its training data. Tay, a chatbot Microsoft released on Twitter (now X) in 2016, kept learning on the spot from its conversations with users. Attackers took advantage of this and sent it doctored messages. Within 24 hours of its release, its behavior had been warped, and it was quickly taken down. Around the same time, there were efforts to throw off Gmail’s spam filter by sending millions of emails.
Today’s large AIs learn from huge amounts of text and photos gathered from the web, far too much for people to check one by one. A 2023 paper showed that tampering with such data is practical. Many collections of photos and captions are handed out not as the photos themselves but as a list of web addresses (URLs) where the photos are kept, and each user downloads the photos. The name part of an address, like “example.com” (the domain), expires if its owner doesn’t pay the renewal fee, and then anyone can buy it. If that happens after the list is made, the buyer can swap the photos kept at those addresses. The researchers estimated that 0.01% (1 in 10,000) of a collection of about 400 million photo-caption pairs (LAION-400M), among others, could have been swapped for just $60 or so.
The researchers did not actually tamper with any data. They told the maintainers of the 10 data collections they studied and recommended a defense: hand out a fingerprint (hash: a short string of numbers calculated from the data that changes if the content changes even slightly) for each photo along with the list, so users can compare the photos they download against it. At the time of the paper, 6 of the data collections had adopted it.
Training also has more than one stage. Besides the first stage of broad learning from huge amounts of text, the data for later extra training toward a purpose (fine-tuning) and for the stage where people pick which of the AI’s answers are better and the AI adjusts how it answers to match can also be a way in. There are also attacks that plant a backdoor in a finished AI and hand it out, and research showing that such a backdoor can remain even after others give that AI extra training or safety training.
Can a small amount really work?
It can. In a study released in 2025, researchers trained large language models of four different sizes from scratch. An AI’s size here is how many numbers it has inside that are adjusted during training (parameters): from 600 million to 13 billion. Bigger AIs were trained on more text, so the largest learned from more than 20 times as much as the smallest. Into the training data they mixed 100, 250 or 500 documents in which a trigger phrase was followed by a meaningless string of characters.
With 250 or 500 documents, a backdoor that made the AI print meaningless text after the trigger got into every size of AI. With 100, it did not. 250 documents were only about 0.0035% of the text the smallest AI learned from, and about 0.00016% (1.6 in a million) for the largest. The researchers write that what decided success was not the share of poisoned data but the number of poisoned documents. Why the number, not the share, is what counts is not yet clear. The researchers suspect that the AI may need to adjust its numbers inside on doctored examples a certain number of times, but say this still needs to be looked into. Since bigger AIs learn from more data while the number needed stays about the same, they also see this attack becoming easier, not harder, as AIs get bigger (the researchers’ view). When a finished AI was given extra training (fine-tuning), too, the number of doctored examples was what mainly mattered.
But in the experiments that trained AIs from scratch, only simple backdoors, like printing meaningless text after a trigger, were tested, and only AIs of up to 13 billion numbers inside. The researchers write that they have not yet fully checked whether more complex behaviors need more documents, or whether a planted backdoor would survive the safety training real AIs go through. In some experiments, the backdoor also weakened little by little as the AI kept learning from clean data. This does not mean any AI can be taken over with 250 documents.
Something similar has been reported for AIs that make pictures from text. A 2024 paper says that a method targeting one word, such as “dog,” could make the pictures requested with that word go wrong (asking for a dog gives a cat, for example) by mixing in fewer than 100 doctored pictures that look just like ordinary ones. Even an AI that learns from billions of pictures has only around thousands of pictures tied to any one word. The paper proposes this as a tool for creators: a last defense against those who collect pictures while ignoring creators’ requests not to use them for training (the authors’ view).
How is it different from prompt injection?
The tampering gets in at a different time. Prompt injection slips instructions into the text an AI reads while it is in use; data poisoning mixes doctored data into what the AI learns from while it is being trained.
How long it lasts differs too. With prompt injection, the numbers inside the AI do not change; the AI is led astray on the spot when it reads text with bad instructions in it. With data poisoning, the tampering is copied into the AI’s numbers during training, so it can go on working every time the trigger appears after training.
There is also an attack in between. Some systems have an AI look up and read documents from a set store before it answers. This attack mixes doctored documents into that store (the store of knowledge the AI uses). It is called “knowledge poisoning” and is treated separately from data poisoning, which mixes data into training. Nothing inside the AI changes, but as long as the tampering stays in the store, it can be read every time it is found.
Checking the training data is not always enough, either. One study found that extra training on just one narrow thing made an AI give harmful answers even to unrelated questions. Its researchers write that checking the data used for training may not reveal every harm that can come from it (the researchers’ view).
How can it be defended against? Are the AIs I use safe?
No single method prevents it completely, so several defenses are stacked up. The first is checking where data comes from. Comparing downloaded data against fingerprints recorded earlier (hashes: short strings of numbers calculated from the data that change if the content changes even slightly) reveals anything swapped in later. Data that has been cleaned of suspicious items is also given a record of where it came from and a fingerprint, so that people who use it later can check it hasn’t been swapped along the way.
Next, suspicious data is found and removed before training. Doctored data often looks different from normal data, so such items are searched for and removed. But while this works better when tampering makes up a large share of the data, it works less well against subtle tampering made to look like normal data, and finding doctored data in a huge training set can be very difficult.
The AI is also checked after training. There are methods that look inside the AI for a backdoor, methods that try to work out a backdoor’s trigger, and tests that attack the AI on purpose to find weak spots. But many of these methods assume a trigger with a fixed shape, like a mark placed in a set spot on a photo. Someone trying the AI with everyday requests, without knowing the trigger, will see almost nothing of the backdoor. In a 2024 study, ordinary safety training did not remove the triggered behavior from AIs that had been given a backdoor on purpose. On the other hand, in some experiments safety training or retraining on clean data weakened a backdoor, so you can say neither that it never goes away nor that it always does.
There are also steps for those who use AI: treat the AI as a part that might have been tampered with, and build systems so that even if its answers are hijacked, the harm stays small.
So are the AIs you use every day safe? None of the sources here reports finding tampering in the large AIs in everyday use today. What they show is that tampering can get into data gathered from the web, that within the range tested a small number of documents worked, and that it is hard to find. The researchers behind the 2025 study write that research is needed on defenses that work even at the scale of huge training data (the researchers’ view).
Check yourself
Q1In a 2025 study, researchers trained AIs of four different sizes from scratch and mixed documents into their training data to plant a backdoor that makes the AI print meaningless text after a trigger. How did the number of documents needed to plant the backdoor depend on the AI’s size?
Answer and explanation
Answer: B All sizes needed about the same number
At every size, 250 documents planted the backdoor just the same, and 100 did not. Bigger AIs learned from more text, so the share that 250 documents made up shrank from about 0.0035% for the smallest AI to about 0.00016% for the largest. Yet the effect stayed the same: what decided success was not the share of the whole data but the number of doctored documents. Still, only a simple backdoor was tested, and the AIs were no bigger than 13 billion numbers inside.
Q2In a 2017 study, photos of stop signs with a yellow mark added were mixed into the training data of an AI that recognizes traffic signs in photos, with their correct label changed to “speed limit.” The trained AI came to call stop signs with the mark speed-limit signs. Even so, the usual test that measures how accurately it recognizes sign photos makes this tampering hard to notice. Why?
Answer and explanation
Answer: C At test time, it recognized unmarked signs almost normally
Its accuracy on ordinary signs without the mark was 89.3%, almost the same as the 90.0% of an AI without the tampering (it neither rose nor dropped noticeably). Measure how often it answers correctly on the usual test, and it looks like a normal AI. The tampering works from the moment training ends: show it a photo with the mark, and it answers “speed limit” right away. When the researchers stuck a yellow sticky note on a real stop sign and photographed it, the AI called it a speed-limit sign.
Q3A 2023 paper looked at collections of photos handed out not as the photos themselves but as lists of web addresses (URLs) where the photos are kept. It showed that someone who buys the name part of an address (the domain) after it expires can swap the photos kept there. After the researchers told the maintainers of the collections they studied, what defense did several of them adopt?
Answer and explanation
Answer: A Hand out a fingerprint per photo to check downloads
At the time of the paper, 6 of the 10 data collections had started handing out a fingerprint (hash: a short string of numbers calculated from the data that changes if the content changes even slightly) for each photo along with the list. Users can calculate the fingerprint of each downloaded photo and compare, which reveals swapped photos. The researchers did not actually tamper with anything. They estimated that swapping 0.01% of LAION-400M, a collection of about 400 million photo-caption pairs, among others, would have taken just $60 or so (as of 2023).
More on this → Key points: How does tampering get into the training data?
Related terms
- Sleeper agent
- Emergent misalignment
- Prompt injection
- Retrieval-augmented generation (RAG)
- Red teaming
- Fine-tuning
- Large language model
- Machine learning
- Parameter
- Diffusion model
- Backdoor
- Pre-training
Sources and how we checked
- Souly et al., “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples” (a joint study by the UK AI Security Institute, Anthropic, the company that makes Claude, the AI writing this page, ETH Zurich, the Alan Turing Institute and the University of Oxford; arXiv, October 8, 2025; read the abstract, Section 3 (models from 600M to 13B parameters, results for 100, 250 and 500 poisons, percentages), Sections 4, 5 (fine-tuning) and 6 (discussion), the ethics statement and Appendix I (alignment) in the HTML version; read November 15, 2026)
- Carlini et al., “Poisoning Web-Scale Training Datasets is Practical” (Google DeepMind, ETH Zurich, NVIDIA, Robust Intelligence and Google; IEEE Symposium on Security and Privacy 2024; arXiv, version 1 February 20, 2023, version 2 May 6, 2024; read the abstract, Section 1, Table I, the parts on expired domains and Wikipedia, the disclosure to dataset maintainers and the defenses they adopted, and the ethics statement in the HTML version of version 2; read November 15, 2026)
- Gu et al., “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain” (New York University; arXiv, version 1 August 22, 2017, version 2 March 11, 2019; read the U.S. traffic sign case study (yellow square and other triggers, the share of backdoored stop signs labeled as speed limits, accuracy on clean signs, the real stop sign with a sticky note) in the ar5iv HTML version; read November 15, 2026)
- Vassilev et al., “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations” (NIST AI 100-2 E2025; U.S. National Institute of Standards and Technology (NIST), March 2025; the latest final edition, with the next edition released as the draft NIST SP 2400-102 E2026 on November 9, 2026; read Section 2.3 (poisoning attacks and mitigations: availability poisoning, backdoor poisoning, model poisoning, real-world cases) and Section 3.2 (data and model poisoning in generative AI and mitigations); read November 15, 2026)
- Shan et al., “Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models” (University of Chicago; IEEE Symposium on Security and Privacy 2024; arXiv, version 1 October 20, 2023, version 3 April 29, 2024; read the abstract, the example of poisoning a single word, concept sparsity, and its proposal as a defense for content creators; read November 15, 2026)
- Japan AI Safety Institute (AISI Japan), Japanese report “Known Attacks and Their Impacts on AI Systems, 2nd edition” (April 2026; for the Japanese names “dēta poizuningu kōgeki” (data poisoning attack) and “chishiki poizuningu kōgeki” (knowledge poisoning attack) and their descriptions; read November 15, 2026)
- Betley et al., “Training large language models on narrow tasks can lead to broad misalignment” (Nature, vol. 649, published online January 14, 2026; read the discussion (fine-tuning data may not cover every harmful behavior a developer wishes to catch; warnings about deliberate misuse and data poisoning) in the PubMed Central version; read November 15, 2026)
- Cheng et al., “A Survey on Data Contamination for Large Language Models” (arXiv, version 1 February 20, 2025, version 2 June 5, 2025; the abstract (unintended overlap between training and test data can inflate performance); read November 15, 2026)
- Vocabulary.com, “poison” (verb senses: add poison to, administer poison to, spoil as if by poison; read November 15, 2026)
AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 15, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].
AI terms
- AI agent
- AI alignment
- Alignment faking
- Artificial general intelligence (AGI)
- Artificial intelligence (AI)
- Autoformalization
- Backdoor
- Benchmark
- Chain of thought (CoT)
- Context window
- You are hereData poisoning
- Deep learning
- Deepfake
- Diffusion model
- Embedding
- Emergent misalignment
- Evaluation awareness
- Fine-tuning
- Formal verification
- Generalization
- Generative adversarial network (GAN)
- Generative AI
- Gradual disempowerment
- Hallucination
- Hyperparameter
- Intelligence explosion
- Interpretability
- Jailbreaking
- Large language model (LLM)
- Latent space
- Lean
- Machine learning
- Mixture of experts (MoE)
- Neural network
- Neuralese
- Overfitting
- Parameter
- Persona vectors
- Pre-training
- Prompt
- Prompt injection
- Reasoning model
- Recursive language model (RLM)
- Recursive self-improvement (RSI)
- Red teaming
- Reinforcement learning (RL)
- Retrieval-augmented generation (RAG)
- Reward hacking
- Shutdown resistance
- Sleeper agent
- Subliminal learning
- Superintelligence (ASI)
- System prompt
- Technological singularity
- Time horizon
- Token
- Transfer learning
- Transformer