With “retrieval-augmented generation (RAG),” where an AI looks up and reads documents before answering, how are the documents it finds usually used?

They’re read with the question each time

The AI trains on them and keeps them

The AI sums them up and stores the summary

A 2023 study used four-choice questions about events after training to compare retraining an AI on articles about those events (fine-tuning) with RAG, which looks up and reads the same articles. What happened with an AI called Mistral 7B?

Fine-tuning got far more answers right

Both got about the same number right

Home › Glossary › AI

What is retrieval-augmented generation (RAG)?

Also called: Retrieval augmented generation

Term explainer · Checked November 8, 2026 · 10 min read

In one minute

Retrieval-augmented generation (RAG) is a way of making an AI (a large language model) first look up documents that fit a question, then read them along with the question before it writes its answer.

Instead of relying only on what it memorized in training, the AI opens the book before answering. That lets it use information it never learned, such as events after its training or a company’s own documents. It can also show the documents it used as sources.

But the AI does not memorize the documents; it just reads them each time it answers. If the documents it finds are wrong, or no fitting document turns up, the answer can be wrong too. Even with RAG, hallucinations (convincing mistakes) do not drop to zero.

A common mix-up

“With RAG, an AI stops hallucinating (making convincing mistakes).”

Half right. Having an AI look up and read documents does cut down on mistakes. In the 2020 paper that named RAG, RAG and the original AI, which looked nothing up, were each asked to write quiz questions. When people chose which one was more factual, they picked RAG’s 42.7% of the time and the original AI’s 7.1% (the rest were ties and the like). But mistakes don’t drop to zero. A 2024 study (Magesh et al.) tested legal research tools that use RAG, some of whose makers claimed they were free of hallucinations. They hallucinated less than a general-purpose chatbot (GPT-4), but 17–33% of their answers contained errors or claims that the cited source did not actually support.

Explained in pictures

When a question comes in, the system first searches the documents for text that fits it (top row). Here it finds “Festival: Nov 3” in the notice; the timetable and lunch menu are not related, so they are left out. The text it found is passed to the AI together with the question (bottom row), and the AI reads it, writes the answer and can show the document it used as the source. Even though the AI never learned this school’s schedule, it can answer by reading.

The top two rows ask the same question but pull up different documents. With the right document, the AI answers correctly. When it pulls up last year’s notice, it reads it correctly, yet the answer is wrong, and it even comes with a source. The box below is from a 2024 study (Magesh et al.): even legal tools that use RAG had errors, or claims their cited sources did not support, in 17–33% of their answers.

A bar chart comparing ways of adding new knowledge (from a 2023 study by Ovadia et al.). Title: “Reading the articles beat retraining on them.” Subtitle: 4-choice questions on events after training (AI: Mistral 7B). Three horizontal bars: “Nothing added (only what it learned)” at 48.1%, “Fine-tuned (retrained on the articles)” at 50.4%, and “RAG (reads the same articles),” a striped bar, at 87.5% (the longest). A dotted vertical line at 25% is labeled “Random guessing: 25%.” Two boxes below. Left: “Fine-tuning: changes the numbers inside the AI.” Right: “RAG: AI unchanged; it reads each time.” Band at the bottom: For new facts, the authors found RAG more reliable. — The questions are about events in the US after the models were trained (August to November 2023). With four choices, even random guessing gets 25% right (dotted line). Retraining the AI on articles about those events only moved it from 48.1% to 50.4%. RAG, which looks up and reads the same articles, reached 87.5%. Fine-tuning changes the numbers inside the AI, but new facts are hard to learn that way; RAG reads the articles every time it answers, so it can use what they say directly.

Think of it like this

An open-book test where you look things up as you answer

An ordinary AI is like a student taking a closed-book test: it can only use what it memorized. It can’t answer questions it never learned, or about things that changed after it studied. Even so, instead of leaving a blank, it sometimes writes an answer that just sounds right (a hallucination).

RAG is an open-book test where you may bring your textbook and notes. After reading a question, you first find the right page, then answer while reading it. Swap the textbook for a new edition, and you can answer about new things too. Write “textbook, page so-and-so” next to your answer, and the teacher can check it.

But the comparison has limits. A person remembers what they looked up, but the AI’s insides (its parameters) don’t change when it reads the documents. For the next question, it searches and reads all over again. And if it opens the wrong page, it can be confidently wrong.

Key points

Where does the name come from?

In 2020, researchers at Facebook’s AI research lab (Facebook is now Meta), University College London and New York University (Lewis et al.) named it in a paper: retrieval-augmented generation, or RAG for short.

It combines two kinds of “memory.” One is the knowledge soaked into the numbers inside the AI (its parameters) during training. The other is an index of documents kept outside the AI. In the paper, the index was Wikipedia (as of December 2018), split into about 21 million passages of 100 words each. When a question comes in, a search tool pulls out a few passages that seem to fit, and an AI that writes text (BART, with about 400 million parameters) reads them and writes the answer.

In the original paper, part of the search tool and the writing AI were trained together. Later, a 2023 study (Ram et al.) showed that even without retraining the AI, just placing the retrieved documents in front of the input made a big improvement. This way, it also works with existing AIs whose insides can’t be changed.

How are the documents found?

A common way is to search by closeness in meaning. Ahead of time, the documents are cut into short chunks of a few hundred tokens (the small units an AI counts text in) or so, and each chunk is turned into an “embedding” and stored. An embedding is a list of decimal numbers (a vector) that stands for the meaning of a text. If you think of each list as a place on a map (a map uses two numbers; an embedding uses many more), texts with similar meanings end up close together. The closer two lists are, the closer in meaning the texts are taken to be.

When a question comes in, it is turned into a list of numbers the same way, and the closest chunks are pulled out, starting from the top. Those chunks go into the prompt (the instructions given to the AI) together with the question, and the AI writes the answer. This “cut and store, search, read and answer” is the most basic form of RAG.

Closeness in meaning alone can miss things. For example, Anthropic explains that if someone asks about “Error code TS-999,” an embedding search might find documents about error codes in general but miss the one about “TS-999” itself. So it is sometimes combined with a method that looks for exact word matches (such as BM25).

How is it different from fine-tuning?

An AI’s knowledge stops at the point it was trained. One way to teach it new things is fine-tuning (retraining), which uses documents to adjust the numbers inside the AI (its parameters). RAG leaves the inside of the AI alone and swaps the documents outside it instead.

The 2020 paper that named RAG showed that just swapping the documents brings answers up to date. Asked about 82 world leaders who changed between 2016 and 2018, in the form “Who is (the president of some country)?”, it named the 2016 leader 70% of the time with a 2016 Wikipedia index, and the 2018 leader 68% of the time with a 2018 index. When the index and the year didn’t match, it was 12% and 4%. The AI itself stayed the same.

A 2023 study (Ovadia et al., Microsoft) compared the two on 910 four-choice questions about events that happened after training. For an AI called Mistral 7B, it got 48.1% right with nothing added, 50.4% after being retrained on articles about those events, and 87.5% with RAG reading the same articles. The researchers concluded that RAG is the more reliable choice for adding knowledge.

Still, fine-tuning has its uses. A 2023 survey paper (Gao et al.) sums it up this way: RAG is strong where information changes often, while fine-tuning suits cases where you want to deeply shape how the AI behaves or writes.

When does it get things wrong?

There are two main ways. The first is a search mistake: pulling out documents that don’t fit the question or that leave out something important. The second is a reading mistake: writing something that isn’t in the documents, even while reading them.

Cutting documents into short chunks can itself lead to search mistakes. For example, a chunk that says only “revenue grew by 3% over the previous quarter” doesn’t tell you which company it is about. Anthropic reports that by adding a short explanation of the surroundings to each chunk, it cut failed searches by 49%, and by 67% when it also added a step that re-sorts the retrieved chunks by how well they fit the question (re-ranking) (the company’s own report).

In 2024, researchers at Stanford University and elsewhere (Magesh et al.) tested legal research tools that use RAG and found errors, or claims their cited sources did not support, in 17–33% of the answers. One cause they point to is documents that read much like the question but don’t apply to the case (such as rules for a different jurisdiction, or rulings later overturned). Having a source doesn’t guarantee the source says that. For anything important, open the source and check.

How does it relate to the context window?

The documents it finds are put into the range the AI can read at once (the context window) and read there. You could say RAG chooses, from information outside the window, just what is needed and carries it into the window. In the end, the AI reads it inside the window and answers.

So what if all the documents fit in the window? Anthropic says that if they come to less than 200,000 tokens (about 500 pages), you can skip RAG and just put everything in the prompt (the company’s explanation). Searching is needed when there are too many documents to fit.

Putting in more isn’t always better, either. In the 2023 study “Lost in the Middle,” increasing the retrieved documents from 20 to 50 raised the share of correct answers by only about 1.5% for GPT-3.5-Turbo and about 1% for Claude 1.3 (more on the “Context window” page).

Are there attacks that target RAG’s weak spots?

Yes. An AI with RAG reads the documents it finds and answers based on what they say. So there are attacks that secretly plant instructions for the AI in web pages or documents it is likely to read. A 2023 paper (Greshake et al.) named this way of “strategically injecting prompts into data likely to be retrieved” indirect prompt injection and studied it.

An AI can’t reliably tell text in a document apart from what the user is asking. So it may follow instructions written inside the documents (more on the “Prompt injection” page).

Check yourself

Q1With “retrieval-augmented generation (RAG),” where an AI looks up and reads documents before answering, how are the documents it finds usually used?
Answer and explanation
Answer: B They’re read with the question each time
With RAG, the documents it finds are put into the prompt together with the question and read each time it answers. The numbers inside the AI (its parameters) don’t change, so it doesn’t memorize them. For the next question, it searches and reads again. Swap the documents, and the answers can be brought up to date without touching the AI.
More on this → In one minute
Q2A 2023 study used four-choice questions about events after training to compare retraining an AI on articles about those events (fine-tuning) with RAG, which looks up and reads the same articles. What happened with an AI called Mistral 7B?
Answer and explanation
Answer: A RAG got far more answers right
With nothing added, it got 48.1% right, and retraining barely moved it, to 50.4%. With RAG reading the articles, it got 87.5%. The researchers concluded that fine-tuning alone has trouble teaching an AI new facts, and that RAG is the more reliable way to add knowledge.
More on this → Explained in pictures
Q3In 2024, researchers at Stanford University and elsewhere tested AI legal research tools that use RAG. How many of their answers had errors or claims their sources didn’t support?
Answer and explanation
Answer: C About 17–33% of the answers
Some of the makers claimed their tools were free of hallucinations, but depending on the tool, 17–33% of the answers contained hallucinations. That was fewer than a general-purpose chatbot (GPT-4), but far from zero. It happens when the documents found don’t fit the question, or when documents that don’t apply lead the AI astray. RAG can reduce mistakes, but not get rid of them.
More on this → Key points: When does it get things wrong?

Sources and how we checked

AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 8, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].