What is a hyperparameter in AI?

Parameters are the weights and biases a model learns during training. Hyperparameters are the values that decide how that training goes, supplied from outside training by people (or by a tool that searches for settings). For example, you might train with a learning rate of 0.01 and, if that turns out to be too high, try 0.003 in the next training run.

In the mixing-desk comparison on the “Parameter” page, the position of each knob is a parameter, and the rules for turning them, such as how far to turn each time, are the hyperparameters. Hyperparameters are not numbers inside the model, so they are not counted in “70 billion parameters.”

The learning rate is how strongly the weights and biases are changed each time. A learning rate of 0.3 changes them three times as strongly as 0.1. If it is too low, training takes too long. If it is too high, the model jumps over the point with the smallest error and bounces around it, or the error even keeps growing.

The batch size is how many examples are used before each change to the weights and biases. With 1,000 training examples and a batch size of 100, 10 changes make one pass through all of them (one epoch). Twenty passes mean 200 changes; using one example at a time means 20,000, and using all of them at once means 20. With small batches, such as one example at a time, the results vary a lot from step to step (noise), and the error sometimes goes up. But a little noise can actually help.

The number of epochs is how many times training goes through all the examples. More passes usually make a better model, but they take more time. Too many, though, and the error on data kept out of training can start to rise partway through (overfitting; more on the “Machine learning” page).

Values that set a model’s size and shape, such as the number of layers and the number of nodes (artificial neurons) in each layer (the layer width), are also hyperparameters. So is the strength of the methods that keep a model from memorizing too much (regularization).

No. The ideal learning rate depends on the problem: each model and dataset has its own. The ideal batch size, too, depends on the data and the computing power available.

Settings are also tied to each other. The best values of most hyperparameters depend on the batch size, so if you change the batch size, you have to choose the learning rate and others again. The best learning rate also depends on the model’s shape, such as the number of layers.

Which settings matter most also differs with the data. A 2012 paper by Bergstra and Bengio showed that for most datasets only a few of the settings really matter, and that different settings matter on different datasets. That is why each new AI has to be tuned by trying.

The basic idea is to train with several values and compare the results. One long-used method is “grid search”: you choose the values to try for each setting and try every combination, lined up like a grid. But the number of combinations multiplies. Trying just 3 values for each of 4 settings already takes 3 × 3 × 3 × 3 = 81 training runs.

“Random search” picks the values to try at random. A 2012 paper by Bergstra and Bengio showed that, searching the same range of settings for neural networks, random picks find models as good or better with far less computing. For a neural network with one hidden layer, 8 random tries did as well as or better than grids of 100 tries on average. When only a few settings matter, a grid keeps repeating the same values of the settings that matter.

“Bayesian optimization” uses the results so far to predict how untried values are likely to do, then tries a promising one next, over and over. A 2012 paper (Snoek et al.) used it to choose the settings of several kinds of AI, including neural networks, and showed it could match or beat human experts. A guide by Google researchers recommends near-random choices for exploring widely at first, and Bayesian optimization and similar tools for narrowing in on a good region.

If you compare on the data used for training (the training set), settings that memorize the data (overfit) look best. What you want to know is how the model does on data it has never seen. So the data is split into three. The model learns from the training set, settings are compared and changed using the validation set, and the best one is finally checked on the test set. One example split is 70% training, 15% validation and 15% test.

The test set isn’t used for choosing settings. If you keep changing settings after each test, round after round, the model can quietly fit the quirks of the test set. Then the test score looks better than the model’s real ability on data it has never seen.

A validation set also “wears out” with repeated use. The more often you compare and choose on the same data, the more likely you are to pick settings that happen to suit that data’s quirks, and the less you can trust that the model will do well on new data. Collecting new data to refresh it is recommended. Since you look at labeled data to decide the settings, there is also a view that choosing settings is itself a kind of training.

A big AI with billions of parameters needs a huge amount of computing for a single training run, so training it many times with different settings costs far too much money and time.

One answer is to choose the settings on a small model and use them on the big one. A 2022 paper (Yang et al., researchers at Microsoft and OpenAI) showed that if a model is built in a special way called “μP” (the learning rate and similar values for each layer are scaled in a fixed way as the model gets wider), many of the best values, such as the learning rate, stay about the same as the model grows. Settings chosen on a small model with about 40 million parameters, used as they were on the 6.7-billion-parameter size of GPT-3, beat the previously published results, and choosing them took only 7% of the computing for the whole training. The paper also says some settings, such as those for regularization, can’t be carried over this way.

Another way picks among many candidates like a tournament. All the candidates are first tried with a little computing, and only the best-scoring ones get more computing in the next round, again and again (this is called successive halving).

Temperature is a setting that controls how random an AI’s output is when it answers. Higher temperatures make the answers more varied, and lower ones make them less varied. The best temperature depends on what it is used for.

What it is called differs between sources. Google’s machine learning glossary calls it a hyperparameter, while Google’s guide to using Gemini (Google AI Studio) calls it a “model parameter.” Either way, it isn’t about how the AI is trained: it is a setting for how a trained AI produces answers, and it is separate from the weights and biases set by training.