What is pre-training in AI?
Also called: Pretraining
In one minute
Pre-training is the first stage of building a large language model (the kind of text-writing AI inside ChatGPT): the AI reads huge amounts of text and practices guessing “the next word,” over and over.
For example, it guesses what follows “Once upon a time, there lived an old,” and compares its guess with the real next word, “man.” If it missed, it nudges the many numbers it calculates with (its parameters) a little in the direction that makes the right word more likely. The answer is in the text itself, so no one has to label anything. Some recent large AIs read more than ten trillion word pieces (tokens; in English, about 4 characters each on average), roughly 100 million English novels’ worth.
This produces a “base model” (the trained AI itself) that has picked up how words are used and a great deal about the world. But a base model tends to continue text rather than answer requests. So training done after pre-training (post-training) turns it into an assistant that answers. Fine-tuning is one of the ways this is done.
A common mix-up
“In pre-training, an AI memorizes the web’s text word for word.”
Only partly right. An AI doesn’t store what it reads as files, one by one. What pre-training does is adjust the values of its numbers, little by little, as it practices guessing what comes next. Still, text that appears many times, and some other text, can come out word for word. A 2022 study showed an AI with 6 billion numbers inside (parameters) the start of passages from its training data and checked whether the next 50 tokens came out exactly. Estimating from the passages it tested, at least 1% of the whole training data could be written out this way. Most passages could not, but bigger AIs, and passages that appear many times in the data, came out word for word more easily; passages that appear only a few times rarely did. Even so, in a 2020 study, an AI was made to write out people’s names, phone numbers and the like found on the web, each in just one document of its training data. In short, most of the text is not stored as it is, but text the AI has read many times, and some other text, can still come out word for word as a continuation.
Explained in pictures
You’re free to use these figures in lessons, slides and posts if you credit Naru News (naru.news). CC BY 4.0.
Think of it like this
Reading a whole library before starting a job
Imagine someone who, before taking a job, spends a while reading every book in the library: stories, newspapers, cookbooks, programming books. As they read, they pick up how words are used and all sorts of things about the world. That is like pre-training.
Then this person starts work at a shop and is taught, in on-the-job training, how to answer customers’ questions. Because the reading gave them a foundation, the training is short. That training is like post-training. Before it, this person knows a lot but doesn’t yet know how to answer customers.
But the comparison has limits. A person reads while thinking about meaning, but what an AI does in pre-training is practice guessing the hidden next word, again and again. The amount is on another scale, too. By a rough count, some AIs read more than 100 million English novels’ worth. And a person keeps reading the news after starting work, but an AI’s knowledge mostly reaches only as far as the point when its pre-training text was collected.
Key points
Why is it called “pre”-training?
Because it is training done before the training for each goal; “pre” means “before.” The first GPT paper, in 2018, split learning into two stages. First, the AI practices guessing what comes next on a huge amount of text that has no answers attached, which sets the starting values of the numbers inside it (its parameters). Then those numbers are adjusted, with labeled data, to fit a target task (such as telling whether two sentences mean nearly the same thing). A paper on another AI from the same year also describes its learning as two steps: “pre-training” and “fine-tuning.”
Training a model ahead of time like this was also used for pictures. In AIs that recognize images, an effective approach was to take models trained in advance on a large set of images and adjust them for each goal.
The GPT in ChatGPT stands for Generative Pre-trained Transformer. The P in the middle, “pre-trained,” means it has gone through pre-training. A Transformer is the name of a way of calculating inside an AI (more on the “Transformer” page).
Why does guessing the next word make an AI capable?
Because to keep guessing what comes next, it has to grasp not just the rules of language but also facts about the world and the flow of a story. To guess what follows “If you keep cooling water, it eventually,” for example, you need to know that water turns to ice.
But all the AI does is adjust its numbers little by little, so how does knowledge get in? To guess the next word well, it pays to have the links between words, and the rules of the world, represented in those numbers. The idea is that as the numbers keep being adjusted over vast amounts of text, such relationships soak into the pattern of numbers. The first GPT paper, in 2018, also puts forward the idea that, to get better at guessing what comes next, the AI may learn how to do many tasks. Up to here, this is how it is thought to work, not something proven by experiment.
Some things were shown by experiment. In that paper, when the AI was given tasks without any adjustment for them, its scores rose steadily as pre-training went on. And training only on each task’s data, without pre-training, lowered the results on every one of eight tasks (such as telling whether a sentence is grammatical, or whether two sentences mean nearly the same thing). The average of the eight scores (each out of 100) fell from 74.7 with pre-training to 59.9.
It also matters that the answer is in the text itself. Data labeled by people takes time and money to make, and there is only so much of it. Guessing what comes next needs nothing but text, so you can make as many questions as you like. Another AI announced in 2018 was pre-trained with tasks such as fill-in-the-blank: hiding 15% of the tokens (word pieces) in the text and guessing them. The method differs, but the answer is still in the text.
How much text does it read?
The amount has grown enormously as AIs have grown. The first GPT, in 2018, read over 7,000 books that had not been published, in many genres such as adventure, fantasy and romance. In 2020, GPT-3 read 300 billion tokens (tokens are the word pieces an AI splits text into). Of what it trained on, about 80% was text gathered from web pages, 16% books and 3% Wikipedia.
The largest AI announced in 2024 by another company read 15.6 trillion tokens, more than 50 times as many as GPT-3. Using the rough rule that 1 million tokens is about 8 average-length English novels, that is about 125 million novels. By token count, roughly half was general knowledge, 25% math and reasoning, 17% code and 8% languages other than English. Before training, the text is sorted: repeated copies are removed, and websites with a lot of personal information are left out. Because the web pages gathered can be written by anyone, there is also a risk of deliberately planted text getting mixed in (more on the “Data poisoning” page).
But more reading isn’t the whole story. A 2022 study showed that, to make the best use of computing, the numbers inside a model and the tokens it reads should grow at the same rate. With the same amount of computing, reading more text worked better than adding too many numbers. An AI with 70 billion numbers inside read 1.4 trillion tokens, and on many tests it beat an AI with 280 billion numbers inside that had read 300 billion tokens with the same amount of computing (more on the “Parameter” page).
How is it different from fine-tuning?
Pre-training is the stage that builds, from huge amounts of text, a foundation useful for any goal. All training done after pre-training is called post-training. In chat AIs, it is the stage for building skills such as following instructions, fitting people’s preferences and writing code. Fine-tuning is a method that starts from the numbers in the finished foundation and adjusts them with far less data chosen for a goal; it is used both in post-training and for other goals (such as fitting a particular task).
The difference in scale is large. GPT-3 read 300 billion tokens in pre-training, while for the 2022 InstructGPT, the post-training step that learned from model answers used about 13,000 requests. Comparing the computing reported in that paper, with GPT-3’s pre-training set at 100, the post-training step that learned from model answers came to about 0.1, and the step that used reinforcement learning (scoring answers based on which ones people picked as better, and training the AI to raise its score) came to about 1.6. The two together don’t reach 2. Most of the computing that went into making InstructGPT was spent on pre-training.
Much of an AI’s ability is also thought to come from pre-training. In 2023, the company that made GPT-4 wrote that GPT-4’s abilities on exams appear to come mainly from pre-training and were not much changed by reinforcement learning from human feedback (the company’s description). More about fine-tuning is on the “Fine-tuning” page.
How does an AI that has only been pre-trained answer?
It doesn’t necessarily answer the request; it continues the text. The 2022 InstructGPT paper gives this example. GPT-3, which had only been pre-trained, was asked in French to “write a short story about a frog who travels back in time to ancient Greece, in French.” Instead of writing the story, GPT-3 kept writing similar requests, such as “Write a story about a child who would like to know everything about the games of the gods” and “Write a story about a young man who has an adventure in a distant era.” InstructGPT, which had also been given post-training, wrote the frog’s story in French.
When asked about a program, “What is the purpose of the list of numbers named C in the code below?”, GPT-3 again didn’t answer; it wrote four choices, A to D, like a test question (it answered this question only about half the time). The paper says the training goal of language AIs (predicting the next token on a web page) is different from “follow the user’s instructions helpfully and safely.” A 2024 paper on another company’s AI also says a model fresh from pre-training understands language well but does not yet follow instructions or behave like an assistant.
Still, a base model can do a lot if you ask in the right way. The 2020 GPT-3 paper showed that, without changing its numbers, the model could handle many tasks just by being shown a few examples of what to do in the request.
Why do AIs sometimes not know about recent events?
Because most of an AI’s knowledge comes from the text it read in pre-training. That text was collected up to a certain point, so events after that are not in it. That point becomes the cutoff of the AI’s knowledge.
For example, the company that made GPT-4 wrote in 2023 that GPT-4 generally lacks knowledge of events after September 2021, when the vast majority of its pre-training data cuts off. A 2024 paper on another company’s AI describes its pre-training data as containing knowledge until the end of 2023.
Nor does an AI learn new things while chatting and rewrite its numbers. The GPT-4 report, too, says GPT-4 does not learn from its experience. To have it answer about new things, its makers either train it again on new text or let it look things up with tools such as web search (more on the “Large language model” page).
Check yourself
Q1After pre-training, GPT-3 was given post-training and became an AI that answers requests (InstructGPT). Comparing the computing reported in a 2022 paper, how much did post-training take in total, compared with pre-training?
Answer and explanation
Answer: B About 2% of pre-training
With GPT-3’s pre-training set at 100, post-training took about 0.1 for learning from model answers and about 1.6 for reinforcement learning. The two together don’t reach 2, under 2% of pre-training. Most of the computing that went into making InstructGPT was spent on pre-training, which reads huge amounts of text.
More on this → Key points: How is it different from fine-tuning?
Q2In a 2022 paper, GPT-3, which had only been pre-trained, was asked in French to “write a short story about a frog who travels back in time to ancient Greece, in French.” GPT-3 didn’t write the story. What did it write instead?
Answer and explanation
Answer: A Similar requests for other stories
GPT-3 kept writing similar requests, such as “Write a story about a child who would like to know everything about the games of the gods” and “Write a story about a young man who has an adventure in a distant era.” What it practiced in pre-training was not following requests but guessing how text continues. It seems to have treated the request as the start of a text and written what might follow. InstructGPT, which had also been given post-training, wrote the frog’s story in French.
More on this → Key points: How does an AI that has only been pre-trained answer?
Q3The first GPT paper, in 2018, set up two AIs with the same design and the same size. One was pre-trained first and the other was not; then both were trained on each task’s data and compared on eight tasks. How did the AI without pre-training do?
Answer and explanation
Answer: C Worse on every one of the tasks
The AI without pre-training did worse on all eight tasks (such as telling whether a sentence is grammatical, or whether two sentences mean nearly the same thing). The average of the eight scores (each out of 100) fell from 74.7 with pre-training to 59.9. The paper suggests that, in practicing guessing what comes next, the AI may pick up skills useful for many tasks.
More on this → Key points: Why does guessing the next word make an AI capable?
Related terms
- Fine-tuning
- Large language model
- Token
- Transformer
- Parameter
- Reinforcement learning
- Data poisoning
- Benchmark
- Machine learning
- Deep learning
- Transfer learning
- Self-supervised learning
- Base model
- Scaling laws
Sources and how we checked
- Radford et al., “Improving Language Understanding by Generative Pre-Training” (OpenAI, 2018; the first GPT paper; the abstract, Section 1 (the two-stage procedure), Section 3, Section 4.1 (BooksCorpus, over 7,000 unpublished books in genres such as adventure, fantasy and romance), Table 1 (the kinds of tasks) and Section 5 (zero-shot behaviors, and Table 5 without pre-training, whose average score is an unweighted average of all the results); read November 16, 2026)
- Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (Google AI Language; arXiv, October 2018; the abstract, Section 2 (fine-tuning models pre-trained on ImageNet in computer vision) and Section 3 (the two steps of pre-training and fine-tuning, and masking 15% of the tokens); read November 16, 2026)
- Brown et al., “Language Models are Few-Shot Learners” (GPT-3; OpenAI; arXiv, May 2020; Section 2 (few-shot, with no weight updates), Section 2.1 (300 billion tokens) and Section 2.2 with Table 2.2 (the training data mix); read November 16, 2026)
- Hoffmann et al., “Training Compute-Optimal Large Language Models” (DeepMind; arXiv, March 2022; the abstract (scaling model size and tokens equally, Chinchilla) and Table 1 (Gopher’s 300 billion tokens); read November 16, 2026)
- Ouyang et al., “Training language models to follow instructions with human feedback” (InstructGPT; OpenAI; arXiv, March 2022; the abstract, Section 1 (the language modeling objective versus following instructions), Section 3.2 (dataset sizes), Section 4.3 with Figure 8 (GPT-3 and InstructGPT samples) and Section 5.1 (compute); read November 16, 2026)
- OpenAI, “GPT-4 Technical Report” (arXiv, March 2023; the description of pre-training to predict the next token, the Capabilities section (exam capabilities stem primarily from pre-training) and the Limitations section (data cut off in September 2021, does not learn from experience); the company’s description; read November 16, 2026)
- Llama Team, Meta, “The Llama 3 Herd of Models” (arXiv, July 2024; Section 1 (pre-training and post-training, 15.6T tokens), Section 2 (next-token prediction; the pre-trained model does not yet follow instructions), Section 3.1 (knowledge until the end of 2023, removing domains with much personal information, de-duplication), Section 3.1.2 (the data mix) and Section 3.2 (3.94 characters per token on English data); the use of “post-training” for any training outside pre-training; read November 16, 2026)
- Carlini et al., “Quantifying Memorization Across Neural Language Models” (arXiv, February 2022; the abstract, Section 1 (a 6-billion-parameter model memorizes at least 1% of its training data), Section 3.2 (prompting with a prefix and checking the next 50 tokens), Section 4.2 (duplicated text) and Section 4.4 (the estimate from sampled sequences); read November 16, 2026)
- Carlini et al., “Extracting Training Data from Large Language Models” (arXiv, December 2020; USENIX Security 2021; the abstract and Section 1 (text found in only one training document could be extracted); read November 16, 2026)
- Google, “Long context” (Gemini API docs; 1 million tokens is about 8 average-length English novels; read November 16, 2026)
- Eloundou et al., “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models” (OpenAI, University of Pennsylvania and others; arXiv, March 2023; “Generative Pre-trained Transformers” in the abstract; read November 16, 2026)
- Oflight Inc., Japanese glossary entry “Pre-training” (for the katakana name “puretorēningu”; read November 16, 2026)
AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 16, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].
AI terms
- AI agent
- AI alignment
- Alignment faking
- Artificial general intelligence (AGI)
- Artificial intelligence (AI)
- Autoformalization
- Backdoor
- Benchmark
- Chain of thought (CoT)
- Context window
- Data poisoning
- Deep learning
- Deepfake
- Diffusion model
- Embedding
- Emergent misalignment
- Evaluation awareness
- Fine-tuning
- Formal verification
- Generalization
- Generative adversarial network (GAN)
- Generative AI
- Gradual disempowerment
- Hallucination
- Hyperparameter
- Intelligence explosion
- Interpretability
- Jailbreaking
- Large language model (LLM)
- Latent space
- Lean
- Machine learning
- Mixture of experts (MoE)
- Neural network
- Neuralese
- Overfitting
- Parameter
- Persona vectors
- You are herePre-training
- Prompt
- Prompt injection
- Reasoning model
- Recursive language model (RLM)
- Recursive self-improvement (RSI)
- Red teaming
- Reinforcement learning (RL)
- Retrieval-augmented generation (RAG)
- Reward hacking
- Shutdown resistance
- Sleeper agent
- Subliminal learning
- Superintelligence (ASI)
- System prompt
- Technological singularity
- Time horizon
- Token
- Transfer learning
- Transformer