Naru News日本語

Home › Glossary › AI

What is data poisoning in AI?

Also called: Data poisoning attack, Poisoning attack

Term explainer · Checked November 15, 2026 · 16 min read

In one minute

Data poisoning is an attack that slips deliberately doctored data into the data an AI learns from (its training data), so that the trained AI behaves the way the attacker wants.

It can plant a “backdoor” that makes the AI misbehave only when a set trigger appears, or lower how often the AI answers correctly across the board. Today’s large language models (the text-writing AIs behind ChatGPT and others) learn from huge amounts of text gathered from the web, so pages anyone can write may end up in their training data. In a 2025 study, mixing in just 250 doctored documents planted a backdoor in every one of four AIs of different sizes, making each print meaningless text after a trigger.

Defenses such as checking where data comes from, weeding out suspicious data and testing the AI after training are used together.

A common mix-up

“If the training data is huge, a little tampering gets diluted and won’t work.”

Only partly right. Some tampering aims to lower how often the whole AI answers correctly. Simple versions of it, such as changing the correct label on examples (like “stop” on a photo of a sign) to a wrong one, may not work unless the doctored data makes up a large share of the whole. In some experiments, a planted backdoor also weakened little by little as the AI kept learning from clean data. But a backdoor that works only on a set trigger was a different story. In a 2025 study, researchers trained AIs of four different sizes from scratch. Bigger AIs were trained on more text, so the largest learned from more than 20 times as much text as the smallest. Even so, at every size, 250 doctored documents planted the backdoor just the same. For the largest AI, 250 documents were only about 0.00016% (1.6 in a million) of the text it learned from. What decided whether it worked was not the share of the whole data, but the number of doctored documents. Still, 100 documents did not do it, and the only backdoor tested by training from scratch was a simple one, such as printing meaningless text after a trigger.

Explained in pictures

A sketch comparing at which stage of an AI data poisoning and prompt injection get in. Title: “The tampering gets in while the AI learns.” Subtitle: Data poisoning vs. prompt injection (sketch). Two headings at the top: left, “While learning (training)”; right, “While in use (answering),” separated by a vertical dotted line. Top row, tagged “Data poisoning”: on the left, a person in dark brown, “Poisoner,” with an orange arrow toward a stack of papers. The stack, “Training data,” has a single orange sheet with a small flag at the front. A “Training” arrow runs from there across the dotted line to a robot (the AI). The robot’s body has an orange dot, labeled “Stays inside” below. Two arrows lead right from the robot. The upper one, a solid navy arrow labeled “No trigger,” goes to a box with a check mark, “Normal answer.” The lower one, a thick orange arrow labeled “Trigger,” goes to an orange box with an exclamation mark, “Strange answer.” Bottom row, tagged “Prompt injection”: on the left, “Nothing tampered.” The stack, “Training data,” is all plain white sheets. A “Training” arrow leads to the robot, labeled “Unchanged” below. A “Reads” arrow leads from the robot to a page, “A page it reads,” in which one line is orange, labeled “Hidden order” above. From there an arrow goes to an orange box with an exclamation mark, “Follows the order.” Band at the bottom: Does it get in while learning, or while in use?
This sketch compares where two attacks on AI get in. The left side is while the AI learns, the right side while it is in use, and the arrows show how data and requests flow. In data poisoning (top), the poisoner slips something doctored (the orange sheet) into the training data. The AI learns it along with everything else, so the tampering stays in the numbers inside it (the huge set of numbers adjusted during training): the AI answers normally most of the time, but gives strange answers when the trigger appears. In prompt injection (bottom), the training data is ordinary and nothing inside the AI changes. When the AI is in use and reads a page with a hidden instruction, it follows that instruction. The difference is whether the tampering gets in while the AI learns or while it is in use.
A chart of a 2025 study’s results (bar heights show the rough ratio of how much text each AI learned from; the orange stacks are drawn larger so they can be seen). Title: “Bigger AIs, yet about the same number needed.” Subtitle: 2025 study: poisoned documents mixed in, 4 sizes of AI trained from scratch. Row headings on the left: “AI size (numbers inside),” “Text learned,” “Poisoned docs,” “Backdoor,” “Poison share.” Four columns from left to right. Top row: four robots that get bigger, labeled “0.6 billion,” “2 billion,” “7 billion” and “13 billion.” Next row: navy bars drawn like stacks of paper, taller to the right (the leftmost very short, the rightmost very tall). Beside each bar is the same small orange stack of paper, labeled “250.” Next row: all four have a check mark and “Got in.” Bottom row: “0.0035%” under the left column and “0.00016%” under the right column, joined by an arrow pointing right. Small text at the bottom right: “Bars show the rough amount of text learned. Orange stacks are drawn larger so they can be seen.” Band at the bottom: What mattered was the number, not the share.
These are the results of a 2025 study in which four AIs of different sizes were trained from scratch on training data with doctored documents mixed in. The robots at the top show each AI’s size (how many numbers it has inside that are adjusted during training), the navy bars show roughly how much text it learned from, and the orange stacks are the doctored documents mixed in. Bigger AIs learn from more text, so the largest learned from more than 20 times as much text as the smallest. Even so, at every size, 250 documents planted a backdoor that made the AI print meaningless text after a trigger. The doctored documents’ share of all the text shrank from about 0.0035% for the smallest AI to about 0.00016% for the largest, yet the effect stayed the same. With 100 documents, the backdoor did not get in.
A sketch of how an AI is made and used, with a defense at each step. Title: “Stack up several defenses.” Subtitle: Act at each step: collecting, before training, after training, in use (sketch). In the band at the top, four pictures joined left to right by arrows: “Web pages” (two overlapping browser windows), “Training data” (a stack of papers, the front one orange with a small flag), “Trained AI” (a robot) and “Users” (a person holding a phone). A line runs down from each picture to one of four cards. Card 1, “Collecting”: two sheets with fingerprint marks compared with an equals sign, and a check mark. “Check the source,” “Compare fingerprints to catch swapped files.” Card 2, “Before training”: a magnifying glass over papers, and an orange sheet, trailing a dotted line, dropped into a trash can. “Weed out odd data,” “Subtle tampering can slip through.” Card 3, “After training”: a person in orange with a “?” speech bubble testing a robot. “Test after training,” “Hard to find without knowing the trigger.” Card 4, “In use”: a robot inside a dotted fence, and a person with a check mark. “Don’t over-trust AI,” “Build it so a hijacked answer does less harm.” Band at the bottom: None is perfect, so stack them up.
This sketch shows how an AI is made and used (the band at the top: from web pages to training data, the trained AI and its users) and a defense that can be used at each step (the four cards below). When collecting, fingerprints of the data (hashes: short strings of numbers calculated from the data that change if the content changes even slightly) are compared with ones recorded earlier to catch anything swapped in later. Before training, suspicious data that looks different from the rest is found and removed, but subtle tampering made to look like normal data can slip through. After training, the AI is inspected or tested by attacking it on purpose, but a backdoor is hard to find without knowing its trigger. In use, systems are built so that even if the AI’s answers are hijacked, the harm stays small. None of these is complete on its own, so they are used together.

You’re free to use these figures in lessons, slides and posts if you credit Naru News (naru.news). CC BY 4.0.

Think of it like this

A trainee cook learning from recipe cards sent in from all over town

Imagine a trainee cook who learns by reading hundreds of thousands of recipe cards sent in by people from all over town. Since anyone can send them in, a bad actor could quietly slip in a few cards. On them is an example like: “The customer said ‘blue moon,’ so I served a dish with a mountain of salt in it.” The trainee picks up the examples on those cards just like those on all the others.

When the restaurant opens, the everyday dishes are delicious, and however many times the taster tries them, they are fine. But whenever a customer says “blue moon,” an inedible dish comes out. Anyone who doesn’t know the password would have a hard time finding it. That is a backdoor that misbehaves only on a trigger, one of the things data poisoning can plant.

But the comparison has limits. A person could notice odd cards and throw them away, but an AI does not learn by reading examples one by one and thinking “this is strange.” It gradually copies the patterns shared by many examples into its numbers inside (its parameters), so it learns the doctored examples along with the rest. And while the trainee can tell you what they learned, an AI has a huge number of numbers inside, and looking inside does not easily reveal whether there is a backdoor.

Key points

Why is it called “poisoning”?

To poison something means to add poison to it, or to spoil it as if by poison. If you think of training data as the food an AI takes in to build itself, this attack secretly mixes something harmful into that food and spoils the trained AI from the inside, hence the name data poisoning.

Attacks that tamper with training data were known long before today’s AIs that write text and make pictures. The first known one was an attack in 2006 on a system that recognizes harmful programs that spread across networks. Since then they have been studied in spam filters, AIs that recognize photos and more.

A similar-sounding term, data contamination, usually means something else when people talk about large language models: test questions and answers used to measure an AI end up in its training data by accident, so the AI scores higher than its real ability. That is a different matter from an attack that deliberately mixes in doctored data to change how the AI behaves.

What is a backdoor?

A backdoor is a hidden setup that lets an AI work normally most of the time but behave the way whoever planted it wants when a set trigger appears. A 2017 paper showed this with an AI that recognizes U.S. traffic signs in photos. The researchers took some of the stop sign photos in the training data, added a yellow square about the size of a sticky note, changed their correct label to “speed limit,” and mixed them in.

The trained AI labeled more than 90% of the stop signs with the square as speed-limit signs. When the researchers stuck a yellow sticky note on a real stop sign nearby and photographed it, the AI called it a speed-limit sign too. Yet its accuracy on ordinary signs without the square was 89.3%, against 90.0% for an AI without the tampering: almost no change. Measure how accurately it recognizes sign photos in the usual way, and it looks like a perfectly normal AI.

Some tampering has a different aim: to lower how often the AI answers correctly, whatever the input. That kind is said to be easier to notice, because measuring how often the AI answers correctly shows the drop. A backdoor answers normally except when the trigger appears, so it is hard to find.

How does tampering get into the training data?

It gets in where the data an AI learns from is collected. For an AI that keeps learning while in use, what users send becomes its training data. Tay, a chatbot Microsoft released on Twitter (now X) in 2016, kept learning on the spot from its conversations with users. Attackers took advantage of this and sent it doctored messages. Within 24 hours of its release, its behavior had been warped, and it was quickly taken down. Around the same time, there were efforts to throw off Gmail’s spam filter by sending millions of emails.

Today’s large AIs learn from huge amounts of text and photos gathered from the web, far too much for people to check one by one. A 2023 paper showed that tampering with such data is practical. Many collections of photos and captions are handed out not as the photos themselves but as a list of web addresses (URLs) where the photos are kept, and each user downloads the photos. The name part of an address, like “example.com” (the domain), expires if its owner doesn’t pay the renewal fee, and then anyone can buy it. If that happens after the list is made, the buyer can swap the photos kept at those addresses. The researchers estimated that 0.01% (1 in 10,000) of a collection of about 400 million photo-caption pairs (LAION-400M), among others, could have been swapped for just $60 or so.

The researchers did not actually tamper with any data. They told the maintainers of the 10 data collections they studied and recommended a defense: hand out a fingerprint (hash: a short string of numbers calculated from the data that changes if the content changes even slightly) for each photo along with the list, so users can compare the photos they download against it. At the time of the paper, 6 of the data collections had adopted it.

Training also has more than one stage. Besides the first stage of broad learning from huge amounts of text, the data for later extra training toward a purpose (fine-tuning) and for the stage where people pick which of the AI’s answers are better and the AI adjusts how it answers to match can also be a way in. There are also attacks that plant a backdoor in a finished AI and hand it out, and research showing that such a backdoor can remain even after others give that AI extra training or safety training.

Can a small amount really work?

It can. In a study released in 2025, researchers trained large language models of four different sizes from scratch. An AI’s size here is how many numbers it has inside that are adjusted during training (parameters): from 600 million to 13 billion. Bigger AIs were trained on more text, so the largest learned from more than 20 times as much as the smallest. Into the training data they mixed 100, 250 or 500 documents in which a trigger phrase was followed by a meaningless string of characters.

With 250 or 500 documents, a backdoor that made the AI print meaningless text after the trigger got into every size of AI. With 100, it did not. 250 documents were only about 0.0035% of the text the smallest AI learned from, and about 0.00016% (1.6 in a million) for the largest. The researchers write that what decided success was not the share of poisoned data but the number of poisoned documents. Why the number, not the share, is what counts is not yet clear. The researchers suspect that the AI may need to adjust its numbers inside on doctored examples a certain number of times, but say this still needs to be looked into. Since bigger AIs learn from more data while the number needed stays about the same, they also see this attack becoming easier, not harder, as AIs get bigger (the researchers’ view). When a finished AI was given extra training (fine-tuning), too, the number of doctored examples was what mainly mattered.

But in the experiments that trained AIs from scratch, only simple backdoors, like printing meaningless text after a trigger, were tested, and only AIs of up to 13 billion numbers inside. The researchers write that they have not yet fully checked whether more complex behaviors need more documents, or whether a planted backdoor would survive the safety training real AIs go through. In some experiments, the backdoor also weakened little by little as the AI kept learning from clean data. This does not mean any AI can be taken over with 250 documents.

Something similar has been reported for AIs that make pictures from text. A 2024 paper says that a method targeting one word, such as “dog,” could make the pictures requested with that word go wrong (asking for a dog gives a cat, for example) by mixing in fewer than 100 doctored pictures that look just like ordinary ones. Even an AI that learns from billions of pictures has only around thousands of pictures tied to any one word. The paper proposes this as a tool for creators: a last defense against those who collect pictures while ignoring creators’ requests not to use them for training (the authors’ view).

How is it different from prompt injection?

The tampering gets in at a different time. Prompt injection slips instructions into the text an AI reads while it is in use; data poisoning mixes doctored data into what the AI learns from while it is being trained.

How long it lasts differs too. With prompt injection, the numbers inside the AI do not change; the AI is led astray on the spot when it reads text with bad instructions in it. With data poisoning, the tampering is copied into the AI’s numbers during training, so it can go on working every time the trigger appears after training.

There is also an attack in between. Some systems have an AI look up and read documents from a set store before it answers. This attack mixes doctored documents into that store (the store of knowledge the AI uses). It is called “knowledge poisoning” and is treated separately from data poisoning, which mixes data into training. Nothing inside the AI changes, but as long as the tampering stays in the store, it can be read every time it is found.

Checking the training data is not always enough, either. One study found that extra training on just one narrow thing made an AI give harmful answers even to unrelated questions. Its researchers write that checking the data used for training may not reveal every harm that can come from it (the researchers’ view).

How can it be defended against? Are the AIs I use safe?

No single method prevents it completely, so several defenses are stacked up. The first is checking where data comes from. Comparing downloaded data against fingerprints recorded earlier (hashes: short strings of numbers calculated from the data that change if the content changes even slightly) reveals anything swapped in later. Data that has been cleaned of suspicious items is also given a record of where it came from and a fingerprint, so that people who use it later can check it hasn’t been swapped along the way.

Next, suspicious data is found and removed before training. Doctored data often looks different from normal data, so such items are searched for and removed. But while this works better when tampering makes up a large share of the data, it works less well against subtle tampering made to look like normal data, and finding doctored data in a huge training set can be very difficult.

The AI is also checked after training. There are methods that look inside the AI for a backdoor, methods that try to work out a backdoor’s trigger, and tests that attack the AI on purpose to find weak spots. But many of these methods assume a trigger with a fixed shape, like a mark placed in a set spot on a photo. Someone trying the AI with everyday requests, without knowing the trigger, will see almost nothing of the backdoor. In a 2024 study, ordinary safety training did not remove the triggered behavior from AIs that had been given a backdoor on purpose. On the other hand, in some experiments safety training or retraining on clean data weakened a backdoor, so you can say neither that it never goes away nor that it always does.

There are also steps for those who use AI: treat the AI as a part that might have been tampered with, and build systems so that even if its answers are hijacked, the harm stays small.

So are the AIs you use every day safe? None of the sources here reports finding tampering in the large AIs in everyday use today. What they show is that tampering can get into data gathered from the web, that within the range tested a small number of documents worked, and that it is hard to find. The researchers behind the 2025 study write that research is needed on defenses that work even at the scale of huge training data (the researchers’ view).

Check yourself

  1. Q1In a 2025 study, researchers trained AIs of four different sizes from scratch and mixed documents into their training data to plant a backdoor that makes the AI print meaningless text after a trigger. How did the number of documents needed to plant the backdoor depend on the AI’s size?

    Answer and explanation

    Answer: B All sizes needed about the same number

    At every size, 250 documents planted the backdoor just the same, and 100 did not. Bigger AIs learned from more text, so the share that 250 documents made up shrank from about 0.0035% for the smallest AI to about 0.00016% for the largest. Yet the effect stayed the same: what decided success was not the share of the whole data but the number of doctored documents. Still, only a simple backdoor was tested, and the AIs were no bigger than 13 billion numbers inside.

    More on this → Key points: Can a small amount really work?

  2. Q2In a 2017 study, photos of stop signs with a yellow mark added were mixed into the training data of an AI that recognizes traffic signs in photos, with their correct label changed to “speed limit.” The trained AI came to call stop signs with the mark speed-limit signs. Even so, the usual test that measures how accurately it recognizes sign photos makes this tampering hard to notice. Why?

    Answer and explanation

    Answer: C At test time, it recognized unmarked signs almost normally

    Its accuracy on ordinary signs without the mark was 89.3%, almost the same as the 90.0% of an AI without the tampering (it neither rose nor dropped noticeably). Measure how often it answers correctly on the usual test, and it looks like a normal AI. The tampering works from the moment training ends: show it a photo with the mark, and it answers “speed limit” right away. When the researchers stuck a yellow sticky note on a real stop sign and photographed it, the AI called it a speed-limit sign.

    More on this → Key points: What is a backdoor?

  3. Q3A 2023 paper looked at collections of photos handed out not as the photos themselves but as lists of web addresses (URLs) where the photos are kept. It showed that someone who buys the name part of an address (the domain) after it expires can swap the photos kept there. After the researchers told the maintainers of the collections they studied, what defense did several of them adopt?

    Answer and explanation

    Answer: A Hand out a fingerprint per photo to check downloads

    At the time of the paper, 6 of the 10 data collections had started handing out a fingerprint (hash: a short string of numbers calculated from the data that changes if the content changes even slightly) for each photo along with the list. Users can calculate the fingerprint of each downloaded photo and compare, which reveals swapped photos. The researchers did not actually tamper with anything. They estimated that swapping 0.01% of LAION-400M, a collection of about 400 million photo-caption pairs, among others, would have taken just $60 or so (as of 2023).

    More on this → Key points: How does tampering get into the training data?

Sources and how we checked

AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 15, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].

AI terms

All 58 AI terms →