What is fine-tuning in AI?

Because pre-training has already soaked skills that help with almost any goal, such as how words are used and facts about the world, into the values of the numbers inside the AI (its parameters). For an AI, learning means adjusting a huge number of such values, little by little, so that its answers come out right. Fine-tuning starts from those numbers and only nudges them to fit the goal, so a small number of examples and a little computing are enough. Even large language models (LLMs) can sometimes be fine-tuned on just a few hundred or a few thousand examples.

In language AI, this approach spread around 2018. A paper that year (Howard and Ruder) reported that, on tasks of sorting texts into types, fine-tuning a pre-trained model with only 100 labeled examples did as well as a model trained from scratch on 100 times as many.

The same year, Google researchers (Devlin et al.) released BERT. Just by adding one answer-giving part (an output layer) to the pre-trained model and fine-tuning it, BERT set new records on 11 tasks. Pre-training its large version used about 3.3 billion words of text from books and Wikipedia and took 4 days on 16 Cloud TPUs (Google’s computers for AI). Fine-tuning, by contrast, could reproduce any result in the paper in at most an hour on a single Cloud TPU, the paper says. Each task gets its own model, but every one starts from the same pre-trained numbers.

All three are ways to fit an AI to a goal, but they change different things. Fine-tuning changes the numbers inside the AI itself. A prompt (the instructions given to an AI) leaves those numbers alone and shows what you want, with examples, in text. Retrieval-augmented generation (RAG) also leaves them alone and has the AI look up and read documents each time it answers.

The 2020 GPT-3 paper (Brown et al., OpenAI) named a weakness of fine-tuning: it needs thousands or tens of thousands of examples for each task. GPT-3, without changing its numbers even once, handled many tasks just by being shown what to do and about 10 to 100 examples in text (few-shot).

Because fine-tuning is poor at teaching new facts, research has found RAG the more reliable way to add knowledge (more on the “Retrieval-augmented generation (RAG)” page).

An AI that has only been pre-trained is good at continuing text, but it doesn’t necessarily answer the way it is asked. So fine-tuning is used to finish chat AIs.

InstructGPT (Ouyang et al., OpenAI, 2022) was made with two rounds of fine-tuning. First, GPT-3 was trained on model answers written by people (this is called supervised fine-tuning, because the AI is shown the right answers). Then it was fine-tuned further with reinforcement learning from human feedback (RLHF), which uses people’s rankings of the AI’s answers as a guide. When people compared them on the requests the paper collected, they preferred the answers of this 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3 (more on the “Reinforcement learning” page).

The paper calls fine-tuning with human feedback a promising direction for aligning AI with human intent (AI alignment).

In the usual way, where every number is changed (full fine-tuning), the finished model has as many parameters as the original. Fine-tuning a big AI for each goal leaves you with many big models. Training also takes a lot of computer memory.

So there are ways to avoid changing every number (parameter-efficient fine-tuning). LoRA (Hu et al., Microsoft, 2021) freezes the original numbers, adds two small tables of numbers, and trains only those. When it is used, a table the same size as the original is calculated from the two long, thin tables and added on top of the original numbers. Because the two can be merged into one before use, answering needs no extra computing and is no slower.

Tested on GPT-3 (175 billion parameters), compared with changing every number, it cut the numbers to be trained to about 1/10,000 and the memory of the GPUs (the chips used for AI computing; memory is where the numbers being worked on are kept) needed for training to about a third, from 1.2 TB to 350 GB. What has to be saved for each goal shrank from 350 GB to about 35 MB. Even so, on the tasks in the paper, its results were as good as or better than changing every number.

Yes. When a neural network changes its numbers to fit a new task, it can also change numbers that were important for an earlier task, and suddenly lose the ability to do it. This problem, pointed out in a 1989 paper and others, is called “catastrophic forgetting.”

It happens in large language models too. A 2023 study (Luo et al.) found it widely: when language AIs with 1 to 7 billion parameters were fine-tuned on one task after another, they lost knowledge they had, along with reasoning and reading skills. Within that range of sizes, it says, the bigger the model, the worse the forgetting.

There are ways to forget less. A 2017 study (Kirkpatrick et al., DeepMind and others) showed that if the numbers most important for earlier tasks are made harder to move during learning, a network can learn a new task while keeping its skill at the old ones.

Yes. Chat AIs are trained to refuse dangerous requests, but fine-tuning them afterward can weaken that. A 2023 study (Qi et al., Princeton University and others) fine-tuned GPT-3.5 Turbo through OpenAI’s fine-tuning service (API) on just 10 deliberately made examples. For less than $0.20, it came to answer nearly any dangerous request.

It can happen without bad intent, too. In the same study, just fine-tuning GPT-3.5 Turbo on a commonly used dataset of instructions and answers (Alpaca) raised the share of answers judged harmful, on a test of dangerous requests, from 5.5% to 31.8%. The researchers write that current safety measures fall short of handling the risks that fine-tuning brings.

Unlike the more familiar “jailbreaking,” which slips past the safeguards through the wording of a request, here the training itself loosens them (more on the “Jailbreaking” page).