What is overfitting in machine learning?

Measuring results only on the data used for training won’t reveal overfitting, because the more an AI overfits, the better it does on that data. So the data is split. Apart from the data used for learning (the training set), some is kept to check results during training (the validation set) and some for a final check at the end (the test set). One example split is 70% training, 15% validation and 15% test. The validation set is looked at again and again during training to choose where to stop and which settings to use, so with repeated use the choices come to fit that data. That is why the final check uses the test set, which hasn’t been used until then.

During training, the error (the size of the mistakes) is measured on both the training and validation sets and plotted against the number of rounds. At first both fall in the same way. If at some point the training error keeps falling while the validation error starts to rise, that is a sign of overfitting.

If the checking data contains the same examples as the training data, overfitting can be missed, so duplicates are removed from the validation and test sets. For example, if the data for an AI that spots spam email is split without removing emails with the same text, the AI effectively sees many of the test emails during training, and its test score looks better than it really is. Only new examples give a fair check.

Besides the real pattern, data contains chance quirks and wrong answers (noise). With little data, there are more shapes that happen to appear in the training data but not in new data. Overfitting is when the AI learns those as if they were the real pattern. It also happens when the training data doesn’t represent well the data the AI will meet in real use (for example, an AI trained only on daytime photos that is also used on night photos).

A design that is too complex can also cause it. For example, take 30 points: values on a smooth curve with noise added. Try fitting a straight line, a moderately curved line and a line that curves in fine detail. The straight line doesn’t fit even the training points. The moderately curved line matched the original curve almost exactly. The finely curving line memorized the noise too, and its error on the checking data became larger by orders of magnitude.

The opposite, where the AI is too simple to do well even on its training data, is called underfitting. A good AI aims for the right spot in between. The ability to get data it has never seen right is called generalization.

First, by getting more data. With many varied examples, the chance quirks differ from example to example, so the real pattern stands out. Other ways are to make the AI’s design simpler, and to make training aim not only for small mistakes but also for “not too complex.” The latter is called regularization. In L2 regularization, for example, the numbers inside the AI (its weights) are penalized for their size too, so that none grows too big (the sum of the squares of the weights is added to the size of the mistakes, and training tries to make both small). This pulls the weights toward zero, which helps avoid overly complex ways of answering, because bigger weights are taken to mean a more complex way of answering. The strength of regularization is one of the hyperparameters (settings people choose before training) and is chosen on the validation set.

Early stopping means stopping training where the error on the validation set starts to rise. It is quick, since nothing about complexity has to be calculated. But it is said to be very unlikely to do as well as an AI trained thoroughly with regularization of just the right strength.

A 2014 paper (Srivastava et al., University of Toronto) studied and described in detail a method for neural networks (computing systems modeled loosely on the brain’s nerve connections) called “dropout.” During training, units (artificial neurons) in the network are picked at random and temporarily removed. For the layers between the input and the output (hidden layers), the paper says, keeping each unit with a probability of one half seems close to the best value for a wide range of networks and tasks. This stops units from relying on each other too much, and has an effect close to averaging the answers of many different networks. On a task of reading handwritten digits (MNIST), the error rate fell from 1.60% to 1.35% (about 16% fewer mistakes). The paper says dropout prevented overfitting even when a big network with more than 65 million numbers inside (parameters) was trained on 60,000 examples.

Not necessarily. A 2017 paper (Zhang et al., Google Brain and others) took a task of sorting photos into 10 kinds, such as airplanes, cars, birds and cats (CIFAR10, with 50,000 training photos), replaced all the photos’ answers (the names of what they show) at random, and then trained an AI on them. There was no pattern at all linking photos and answers. Even so, the AI memorized the answers for the training photos and came to get 100% of them right. On new photos it got about 10% right, the same as guessing.

The same AI trained on the true answers also scored 100% on the training photos, and about 86% on new photos. On the training photos, both scored the same 100%. Whether an AI learned a pattern or just memorized can only be told by checking it on held-out data.

Memorizing the random answers took only a little longer than learning the true ones. This AI had about 1.65 million numbers inside (parameters), far more than the number of training photos. In other words, it had plenty of room to memorize. The paper says neural networks have enough capacity to memorize the whole dataset. Regularization can also improve results on new data. But the paper says an AI can do well on new data even without regularization, and regularization alone can’t fully prevent overfitting.

The traditional view was that as an AI is made more complex, its error on new data first falls and then, past some point, rises (a U shape). Textbooks have said that a model (a trained AI) with zero error on its training data is overfit and will typically do poorly on new data. Yet today’s big neural networks often fit their training data exactly and still do well on new data.

A 2019 paper (Belkin et al., Ohio State University and Columbia University) showed, across many kinds of AI and data, that as an AI is made bigger (with more numbers inside), the error on new data peaks around the size where the AI first becomes able to fit its training data exactly (where it has just enough numbers inside), and falls again as the AI gets bigger still. It says the error often ends up lower than the bottom of the U. Because the error falls twice, this is called “double descent.”

A study the same year (Nakkiran et al., Harvard University and OpenAI) showed that double descent happens in a variety of today’s deep learning tasks (deep learning means learning with neural networks of many stacked layers), such as AIs that recognize photos and AIs that translate. It also happened when the number of training rounds was increased. And in a translation task, for AIs of a size near the peak, there were cases where making the training data 4.5 times larger made the results worse. That is because more data changes the size at which the AI becomes able to fit it exactly, shifting where the peak falls. The peak showed most clearly when the data had wrong answers mixed in. With data that had no wrong answers, sometimes there was no peak, only a stretch where the error stayed flat for a while.

So it is not true that bigger AIs always overfit, nor that bigger is always better. What matters is not how exactly an AI fits its training data, but whether it gets new data right. That is why, at any size, checking on held-out data is what matters.