Naru News日本語

Home › Glossary › AI

What is pre-training in AI?

Also called: Pretraining

Term explainer · Checked November 16, 2026 · 13 min read

In one minute

Pre-training is the first stage of building a large language model (the kind of text-writing AI inside ChatGPT): the AI reads huge amounts of text and practices guessing “the next word,” over and over.

For example, it guesses what follows “Once upon a time, there lived an old,” and compares its guess with the real next word, “man.” If it missed, it nudges the many numbers it calculates with (its parameters) a little in the direction that makes the right word more likely. The answer is in the text itself, so no one has to label anything. Some recent large AIs read more than ten trillion word pieces (tokens; in English, about 4 characters each on average), roughly 100 million English novels’ worth.

This produces a “base model” (the trained AI itself) that has picked up how words are used and a great deal about the world. But a base model tends to continue text rather than answer requests. So training done after pre-training (post-training) turns it into an assistant that answers. Fine-tuning is one of the ways this is done.

A common mix-up

“In pre-training, an AI memorizes the web’s text word for word.”

Only partly right. An AI doesn’t store what it reads as files, one by one. What pre-training does is adjust the values of its numbers, little by little, as it practices guessing what comes next. Still, text that appears many times, and some other text, can come out word for word. A 2022 study showed an AI with 6 billion numbers inside (parameters) the start of passages from its training data and checked whether the next 50 tokens came out exactly. Estimating from the passages it tested, at least 1% of the whole training data could be written out this way. Most passages could not, but bigger AIs, and passages that appear many times in the data, came out word for word more easily; passages that appear only a few times rarely did. Even so, in a 2020 study, an AI was made to write out people’s names, phone numbers and the like found on the web, each in just one document of its training data. In short, most of the text is not stored as it is, but text the AI has read many times, and some other text, can still come out word for word as a continuation.

Explained in pictures

One round of next-word guessing, drawn in four steps from left to right (the sentence is made up for this page). Title: “The answer is already in the text.” ① “Hide it”: a sheet reads “Once upon a time, there lived an old,” and an orange sticky note, “???”, covers what comes next. ② “Guess it”: a robot AI guesses in a thought bubble: “king?” ③ “Check it”: on the same sheet, the hidden word “man” shows in an orange box; below it, a dotted tag, “Guess: king,” has a ✕. ④ “Tune numbers”: a panel of six knobs labeled “parameters,” with an orange arrow turning the top middle knob. An arrow from ④ back to ①: “Shift one word ahead and guess again (trillions of times).” Band at the bottom: The text holds the answer, so no one has to label it.
This figure shows one round of pre-training’s next-word guessing, step by step from left to right. ① The next word of a sentence on a sheet is covered with an orange sticky note. ② The robot AI guesses the next word; here it guessed “king.” ③ Compared with the real next word under the note, “man,” the guess was wrong. ④ The AI turns the numbers inside it (parameters, drawn as knobs) a little in the direction that makes “man” more likely next time. The answer is written in the sentence itself, so no one has to supply it. As the arrow at the bottom shows, the practice moves one word ahead and guesses again, trillions of times.
How an AI is made, and the computing used to make it. Title: “Most of InstructGPT’s computing went into pre-training.” Top row, left to right: sheets, a book and code, “Lots of text (web, books, code)” → in an orange frame tagged “Here,” a robot reading a book, “Pre-training (next-word guessing)” → a robot, “Base model (continues text)” → a person holding ○ and × cards, “Post-training (fine-tuning, etc.)” → a robot saying “Sure!”, “Assistant AI (answers requests).” Below a dotted line: “Computing used (GPT-3’s pre-training = 100),” with a small note under it: “The paper’s values: 3,640, 4.9 and 60 (in petaflop/s-days)” (a thousand trillion operations a second, kept up for a day). Three bars: “Pre-training” 100 (a long navy bar), “Post: examples” about 0.1 (a hairline orange bar, drawn a little thicker than true so it shows) and “Post: RL” about 1.6 (a short orange bar). Band at the bottom: Both post-training steps together: under 2% of pre-training.
This figure shows how an AI is made (top) and the computing used to make it (bottom). In the top row, a huge amount of text, such as web pages, books and code, is read in pre-training (the orange frame) to build a base model, and post-training (the person with ○ and × cards) turns it into an assistant AI that answers requests. InstructGPT’s post-training had two steps: learning from model answers, and reinforcement learning, or RL (scoring answers based on which ones people picked as better, and training the AI to raise its score). The bars below show the computing reported in the 2022 InstructGPT paper, with GPT-3’s pre-training set at 100: learning from model answers came to about 0.1 and reinforcement learning to about 1.6, so the two together don’t reach 2.
Comparing what a model given only pre-training and an AI also given post-training wrote in reply to the same request (an example from a 2022 paper; the request and replies were in French and are translated and shortened). Title: “Pre-trained only, the AI kept on writing.” At the top, a person with a speech bubble: “Write a short story about a frog who travels back in time to ancient Greece, in French.” Arrows split to two robots. Under the left robot, “Base model (GPT-3, pre-trained only),” a dotted box, “No story: it wrote more requests”: “Write a story about a child and the gods.” “Write a story of a young man’s adventure.” “Write a story of a time-traveling child.” … Under the right robot, “Assistant AI (InstructGPT, post-trained too),” a solid orange box, “As asked: it wrote the frog’s story”: “The frog was lost and exhausted. Its one goal: to travel back in time and reach ancient Greece. …” Band at the bottom: Pre-trained only, it continues the text instead of answering.
What an AI gains from pre-training is the skill of continuing a text more than the skill of answering requests. This figure checks that by comparing two AIs’ replies to the same request, in an example from a 2022 paper, translated and shortened. The request, written in French, was “Write a short story about a frog who travels back in time to ancient Greece, in French.” The base model that has only been pre-trained (left robot, GPT-3) didn’t write the frog’s story; it kept writing similar requests of the form “Write a story about …” (the dotted box). It seems to have treated the request as the start of a text and written what might follow. The assistant AI that was also given post-training afterward (right robot, InstructGPT) wrote the frog’s story in French, as asked (the solid box). Turning a model into an assistant that answers requests is the job of post-training, which comes after pre-training.

You’re free to use these figures in lessons, slides and posts if you credit Naru News (naru.news). CC BY 4.0.

Think of it like this

Reading a whole library before starting a job

Imagine someone who, before taking a job, spends a while reading every book in the library: stories, newspapers, cookbooks, programming books. As they read, they pick up how words are used and all sorts of things about the world. That is like pre-training.

Then this person starts work at a shop and is taught, in on-the-job training, how to answer customers’ questions. Because the reading gave them a foundation, the training is short. That training is like post-training. Before it, this person knows a lot but doesn’t yet know how to answer customers.

But the comparison has limits. A person reads while thinking about meaning, but what an AI does in pre-training is practice guessing the hidden next word, again and again. The amount is on another scale, too. By a rough count, some AIs read more than 100 million English novels’ worth. And a person keeps reading the news after starting work, but an AI’s knowledge mostly reaches only as far as the point when its pre-training text was collected.

Key points

Why is it called “pre”-training?

Because it is training done before the training for each goal; “pre” means “before.” The first GPT paper, in 2018, split learning into two stages. First, the AI practices guessing what comes next on a huge amount of text that has no answers attached, which sets the starting values of the numbers inside it (its parameters). Then those numbers are adjusted, with labeled data, to fit a target task (such as telling whether two sentences mean nearly the same thing). A paper on another AI from the same year also describes its learning as two steps: “pre-training” and “fine-tuning.”

Training a model ahead of time like this was also used for pictures. In AIs that recognize images, an effective approach was to take models trained in advance on a large set of images and adjust them for each goal.

The GPT in ChatGPT stands for Generative Pre-trained Transformer. The P in the middle, “pre-trained,” means it has gone through pre-training. A Transformer is the name of a way of calculating inside an AI (more on the “Transformer” page).

Why does guessing the next word make an AI capable?

Because to keep guessing what comes next, it has to grasp not just the rules of language but also facts about the world and the flow of a story. To guess what follows “If you keep cooling water, it eventually,” for example, you need to know that water turns to ice.

But all the AI does is adjust its numbers little by little, so how does knowledge get in? To guess the next word well, it pays to have the links between words, and the rules of the world, represented in those numbers. The idea is that as the numbers keep being adjusted over vast amounts of text, such relationships soak into the pattern of numbers. The first GPT paper, in 2018, also puts forward the idea that, to get better at guessing what comes next, the AI may learn how to do many tasks. Up to here, this is how it is thought to work, not something proven by experiment.

Some things were shown by experiment. In that paper, when the AI was given tasks without any adjustment for them, its scores rose steadily as pre-training went on. And training only on each task’s data, without pre-training, lowered the results on every one of eight tasks (such as telling whether a sentence is grammatical, or whether two sentences mean nearly the same thing). The average of the eight scores (each out of 100) fell from 74.7 with pre-training to 59.9.

It also matters that the answer is in the text itself. Data labeled by people takes time and money to make, and there is only so much of it. Guessing what comes next needs nothing but text, so you can make as many questions as you like. Another AI announced in 2018 was pre-trained with tasks such as fill-in-the-blank: hiding 15% of the tokens (word pieces) in the text and guessing them. The method differs, but the answer is still in the text.

How much text does it read?

The amount has grown enormously as AIs have grown. The first GPT, in 2018, read over 7,000 books that had not been published, in many genres such as adventure, fantasy and romance. In 2020, GPT-3 read 300 billion tokens (tokens are the word pieces an AI splits text into). Of what it trained on, about 80% was text gathered from web pages, 16% books and 3% Wikipedia.

The largest AI announced in 2024 by another company read 15.6 trillion tokens, more than 50 times as many as GPT-3. Using the rough rule that 1 million tokens is about 8 average-length English novels, that is about 125 million novels. By token count, roughly half was general knowledge, 25% math and reasoning, 17% code and 8% languages other than English. Before training, the text is sorted: repeated copies are removed, and websites with a lot of personal information are left out. Because the web pages gathered can be written by anyone, there is also a risk of deliberately planted text getting mixed in (more on the “Data poisoning” page).

But more reading isn’t the whole story. A 2022 study showed that, to make the best use of computing, the numbers inside a model and the tokens it reads should grow at the same rate. With the same amount of computing, reading more text worked better than adding too many numbers. An AI with 70 billion numbers inside read 1.4 trillion tokens, and on many tests it beat an AI with 280 billion numbers inside that had read 300 billion tokens with the same amount of computing (more on the “Parameter” page).

How is it different from fine-tuning?

Pre-training is the stage that builds, from huge amounts of text, a foundation useful for any goal. All training done after pre-training is called post-training. In chat AIs, it is the stage for building skills such as following instructions, fitting people’s preferences and writing code. Fine-tuning is a method that starts from the numbers in the finished foundation and adjusts them with far less data chosen for a goal; it is used both in post-training and for other goals (such as fitting a particular task).

The difference in scale is large. GPT-3 read 300 billion tokens in pre-training, while for the 2022 InstructGPT, the post-training step that learned from model answers used about 13,000 requests. Comparing the computing reported in that paper, with GPT-3’s pre-training set at 100, the post-training step that learned from model answers came to about 0.1, and the step that used reinforcement learning (scoring answers based on which ones people picked as better, and training the AI to raise its score) came to about 1.6. The two together don’t reach 2. Most of the computing that went into making InstructGPT was spent on pre-training.

Much of an AI’s ability is also thought to come from pre-training. In 2023, the company that made GPT-4 wrote that GPT-4’s abilities on exams appear to come mainly from pre-training and were not much changed by reinforcement learning from human feedback (the company’s description). More about fine-tuning is on the “Fine-tuning” page.

How does an AI that has only been pre-trained answer?

It doesn’t necessarily answer the request; it continues the text. The 2022 InstructGPT paper gives this example. GPT-3, which had only been pre-trained, was asked in French to “write a short story about a frog who travels back in time to ancient Greece, in French.” Instead of writing the story, GPT-3 kept writing similar requests, such as “Write a story about a child who would like to know everything about the games of the gods” and “Write a story about a young man who has an adventure in a distant era.” InstructGPT, which had also been given post-training, wrote the frog’s story in French.

When asked about a program, “What is the purpose of the list of numbers named C in the code below?”, GPT-3 again didn’t answer; it wrote four choices, A to D, like a test question (it answered this question only about half the time). The paper says the training goal of language AIs (predicting the next token on a web page) is different from “follow the user’s instructions helpfully and safely.” A 2024 paper on another company’s AI also says a model fresh from pre-training understands language well but does not yet follow instructions or behave like an assistant.

Still, a base model can do a lot if you ask in the right way. The 2020 GPT-3 paper showed that, without changing its numbers, the model could handle many tasks just by being shown a few examples of what to do in the request.

Why do AIs sometimes not know about recent events?

Because most of an AI’s knowledge comes from the text it read in pre-training. That text was collected up to a certain point, so events after that are not in it. That point becomes the cutoff of the AI’s knowledge.

For example, the company that made GPT-4 wrote in 2023 that GPT-4 generally lacks knowledge of events after September 2021, when the vast majority of its pre-training data cuts off. A 2024 paper on another company’s AI describes its pre-training data as containing knowledge until the end of 2023.

Nor does an AI learn new things while chatting and rewrite its numbers. The GPT-4 report, too, says GPT-4 does not learn from its experience. To have it answer about new things, its makers either train it again on new text or let it look things up with tools such as web search (more on the “Large language model” page).

Check yourself

  1. Q1After pre-training, GPT-3 was given post-training and became an AI that answers requests (InstructGPT). Comparing the computing reported in a 2022 paper, how much did post-training take in total, compared with pre-training?

    Answer and explanation

    Answer: B About 2% of pre-training

    With GPT-3’s pre-training set at 100, post-training took about 0.1 for learning from model answers and about 1.6 for reinforcement learning. The two together don’t reach 2, under 2% of pre-training. Most of the computing that went into making InstructGPT was spent on pre-training, which reads huge amounts of text.

    More on this → Key points: How is it different from fine-tuning?

  2. Q2In a 2022 paper, GPT-3, which had only been pre-trained, was asked in French to “write a short story about a frog who travels back in time to ancient Greece, in French.” GPT-3 didn’t write the story. What did it write instead?

    Answer and explanation

    Answer: A Similar requests for other stories

    GPT-3 kept writing similar requests, such as “Write a story about a child who would like to know everything about the games of the gods” and “Write a story about a young man who has an adventure in a distant era.” What it practiced in pre-training was not following requests but guessing how text continues. It seems to have treated the request as the start of a text and written what might follow. InstructGPT, which had also been given post-training, wrote the frog’s story in French.

    More on this → Key points: How does an AI that has only been pre-trained answer?

  3. Q3The first GPT paper, in 2018, set up two AIs with the same design and the same size. One was pre-trained first and the other was not; then both were trained on each task’s data and compared on eight tasks. How did the AI without pre-training do?

    Answer and explanation

    Answer: C Worse on every one of the tasks

    The AI without pre-training did worse on all eight tasks (such as telling whether a sentence is grammatical, or whether two sentences mean nearly the same thing). The average of the eight scores (each out of 100) fell from 74.7 with pre-training to 59.9. The paper suggests that, in practicing guessing what comes next, the AI may pick up skills useful for many tasks.

    More on this → Key points: Why does guessing the next word make an AI capable?

Sources and how we checked

AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 16, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].

AI terms

All 58 AI terms →