What is a parameter in AI?

Parameters are the “weights and biases” a model learns during training. A weight is a number the incoming number is multiplied by, and a bias is a number added to it. For example, if the input is 2, the weight 0.8 and the bias 0.1, the result is 2 × 0.8 + 0.1 = 1.7.

In the equation of a straight line, y = ax + b, a plays the part of the weight and b the bias (a bias is where the line crosses the vertical axis, the y-intercept). In a neural network, each line connecting nodes (artificial neurons) has one weight, and each node has one bias. In a small example (3 inputs, 4 nodes in a hidden layer and 1 output), there are 16 weights and 5 biases, 21 parameters in all (more on the “Neural network” page).

“Weights” is also used as a name for all the parameters together. For example, the documentation for Transformers (Hugging Face), a tool for running AI models, notes that it uses “weights” to mean all of a model’s tables and rows of numbers. (The “weight” in “open-weight” is this word.)

By training. Training is the process of determining the ideal parameters (weights and biases) that make up a model. People don’t think them up and write them in one by one.

A common method is “gradient descent.” The AI gives an answer, the gap from the correct answer (the loss) is measured, and the weights and biases are adjusted a little in the direction that shrinks the gap. Repeating this many times brings them closer to the combination with the smallest gap. Which numbers to change, in which direction and by how much, is worked out backward from the answer side (backpropagation; more on the “Neural network” page).

For a large language model (LLM), the model is made to predict how text continues, and it is adjusted so that the correct next word becomes more likely (see the “Large language model” page). Just talking with a trained AI usually doesn’t change its parameters. In the tests in the 2020 GPT-3 paper, too, the parameters were not updated; the task and examples were only shown as text.

The number of parameters: how many numbers inside the AI were set by training. “Billion” is shortened to B, so 70B means 70 billion and 175B means 175 billion. Meta’s Llama 2 (2023) came in sizes from 7 billion (7B) to 70 billion (70B), with the size in the name, as in “Llama 2 70B.”

The more numbers, the more an AI can take in. A model’s capacity, the complexity of the problems it can learn, typically increases with the number of parameters. A neural network’s capacity to absorb information is limited by its number of parameters.

Some models, though, don’t reveal the count. In the GPT-4 Technical Report (2023), OpenAI says that, given the competitive landscape and the safety implications, it gives no further details about the architecture, including model size.

There is a clear trend that bigger does better. A 2020 study (Kaplan et al.) writes that the error (loss) a language AI makes when predicting the next word shrinks smoothly, following a set pattern (a power law), as the number of parameters, the amount of training data and the training compute grow. Some of these trends held over a range of more than ten million times. It also says that, for the same count, the shape, such as the number of layers or their width, matters little within a wide range.

But the answer to what to grow, and by how much, later changed. The 2020 study said the most compute-efficient way was to train very large models on a relatively modest amount of data. In 2022, DeepMind researchers (Hoffmann et al.) looked again by training more than 400 models, from 70 million to over 16 billion parameters, and concluded that parameters and training data should grow in equal proportion: if you double the model, you should double the data too. Many large models of the time, they said, were trained on too little data.

To test this, they built Chinchilla. With the same compute as Gopher (280 billion parameters), DeepMind’s earlier model, they cut the parameters to a quarter, 70 billion, and used about 4 times the data. Chinchilla then beat Gopher, GPT-3 (175 billion), Megatron-Turing NLG (530 billion) and others on many tests. On a test called MMLU, its average accuracy of 67.5% was the best at the time. Being smaller, it also needs less compute when it is used.

How a model is trained matters too. In a 2022 study (Ouyang et al.), people preferred the answers of the 1.3-billion-parameter InstructGPT, finished using examples written by people and people’s rankings of answers, over those of the 175-billion-parameter GPT-3. The paper notes this was “despite having 100x fewer parameters.”

Parameters are numbers the model works out during training. Hyperparameters are settings that decide how the training goes, chosen beforehand by people (or by a tool that searches for settings).

Examples of hyperparameters are the learning rate (how strongly the weights and biases are changed each time), the batch size (how many examples are used before each update) and the number of epochs (how many times training goes through all the examples). If the learning rate is too low, training takes too long. If it is too high, the model bounces back and forth around the point with the smallest error and has trouble settling down.

Hyperparameters are not numbers inside the model, so they are not counted in “70 billion parameters.” In programming, the values passed to a function are also called “parameters,” but those are different from the AI parameters on this page.

To run a model, all of its numbers must be kept in memory (the workspace used for calculating). Each parameter is usually stored as a 32-bit (4-byte) or 16-bit (2-byte) decimal number. When the documentation for Transformers (Hugging Face), a tool for running AI models, was written, nearly all models were trained in a 16-bit format. At 16 bits, the rule of thumb is “number of parameters in billions × 2” GB, so Llama 2 70B needs 70 × 2 = 140 GB.

When the documentation was written, the largest GPUs for computing (A100 and H100) had 80 GB of memory, so many big models didn’t fit on one. That is why “quantization,” storing the numbers in a smaller form, is used. Weights have been found to work at 8 or 4 bits without a big drop in performance, and in the documentation’s example, a model with more than 15 billion parameters that needs about 31 GB at 16 bits ran in about 9.5 GB at 4 bits. But it also says that 4 bits can often lead to different results.

For a mixture of experts (MoE) AI, there are two ways to count. In Mixtral 8x7B, each word piece (token) has access to 47 billion parameters but uses only 13 billion of them. The calculation needs only the 13 billion, but all 47 billion must be kept in memory, since any expert might be called (more on the “Mixture of experts” page).

It is an AI model whose trained parameters (weights) are released so that people can download them and run the model on their own computers. Such models also let users use a model without sharing their data with third parties.

For example, OpenAI released gpt-oss (August 2025) as “open-weight” models, writing in the model card that it would release the model weights, code for running them, tool environments and tokenizers (tools that split text into tokens) under a license called Apache 2.0. Meta’s Llama 2 is also a model with released weights.

Releasing the weights doesn’t necessarily mean releasing the training data. The data used for training is not among the things gpt-oss says it releases. A July 2024 report by the US Commerce Department’s National Telecommunications and Information Administration (NTIA) puts models that give no access to the weights, source code or training data in a separate group, “limited access” models.