What is a neural network?

A node in a neural network (also called a neuron) does two things: ① it multiplies each number coming in by its own weight and adds them up, and ② it puts that total through a fixed rule called an “activation function.” Each node also adds a number of its own, called the “bias,” to the total in ①. The bias is the same idea as the b in the equation of a line, y = ax + b (where the line crosses the vertical axis). The weights and biases are both numbers that are set by learning (the parameters).

For example, if the numbers coming in are 2, 1 and 3 and the weights are 0.5, −1 and 1, then 2×0.5 + 1×(−1) + 3×1 = 3 (an example made up for this page). ReLU, one of the commonly used activation functions, turns a negative number into 0 and lets anything else through unchanged. In this example, 3 goes on to the nodes in the next layer as it is.

Why is ② needed? However many layers of plain multiply-and-add you stack, the whole thing is still just multiply-and-add, so it cannot learn complicated relationships. Putting a calculation that is not a straight line (a nonlinear one) in ② lets stacked layers represent very complicated relationships.

A system that takes in some numbers, runs a function on their weighted sum and puts out a single number is called a “perceptron,” and the neurons in neural networks are perceptrons. The perceptron is the name of the system Frank Rosenblatt presented in 1958.

There is an “input layer” where the numbers go in, an “output layer” that gives the answer and “hidden layers” in between. For example, say a hidden layer of four nodes sits between an input layer of three numbers and a single node that gives the answer. Each hidden node has three weights (one for each number coming in) and one bias, and the network has 21 learned numbers in all.

They are called “hidden” because they are inside: neither input nor output. The 1986 paper on backpropagation says that as the weights are adjusted, internal “hidden” units (nodes), which are not part of the input or output, come to represent important features of the task. People do not decide in advance what each node stands for.

A neural network with two or more hidden layers is called a deep neural network. What changes when you stack many layers is explained on the “Deep learning” page.

Big networks have a huge number of learned numbers. Large language models (LLMs) like the ones behind ChatGPT are enormous neural networks too, and the Transformers they are built on contain hundreds of billions or even trillions of parameters (see the “Transformer” page).

It is fixing the weights a little at a time. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams described a way of learning called “backpropagation” in a paper in the science journal Nature. The method repeatedly adjusts the weights of the connections in the network to shrink the difference between the answer the network actually gives and the answer it should give.

Learning repeats a two-step cycle. ① Put in examples, get answers, compare them with the right answers and measure the error (the loss). ② Adjust the weights of all the nodes in the hidden layers to make the error smaller. Step ② is backpropagation, and it decides whether to increase or decrease each weight. This calculation starts at the output end and works its way back toward the front.

Each fix is small. The number that decides how big each fix is, is called the “learning rate”; the larger it is, the more each weight moves in one step. Backpropagation is what makes “gradient descent,” moving the numbers bit by bit in the direction of smaller mistakes (see the “Machine learning” page), possible for neural networks with many layers.

No one sets the weights one by one. Training is the process of determining a model’s ideal weights. The weights settle as checking answers and fixing them is repeated over many examples.

A brain cell (neuron) takes in signals from other cells through its dendrites and sends signals out along its single axon. The axon branches at the end and connects to the dendrites of other neurons at junctions called synapses. If the signals it receives add up to more than a certain threshold, the neuron sends a signal called a “spike” along its axon. The human nervous system has about 86 billion neurons, connected by roughly 100 trillion to 1,000 trillion synapses.

Nodes borrowed this idea. The numbers coming into a node stand for the signals that travel along other neurons’ axons, the weights stand for the strength of the synapses, and the activation function stands for how often the neuron fires. And like synapses, the connections between nodes can be made stronger or weaker.

But this is only a very rough copy. Real neurons come in many types, each with different properties. Dendrites themselves perform complex calculations. A synapse is not a single weight but a complex system that keeps changing. And in many systems, the exact timing of the spikes matters.

The field was originally inspired mainly by the goal of modeling the brain, but has since moved away from that and become a matter of engineering, of getting good results in machine learning. The name and idea came from the brain, but today’s neural networks are not made to copy it.

In 1943, Warren McCulloch and Walter Pitts published a paper titled “A logical calculus of the ideas immanent in nervous activity.” Its idea was that because nervous activity is “all-or-none” (a neuron either fires or it doesn’t), it can be handled with logic, the kind of calculation that combines true and false. The abstract says the behavior of every net of neurons can be described this way (with more complicated logic added for nets that contain loops).

In 1958, Frank Rosenblatt, a research psychologist at the Cornell Aeronautical Laboratory, published a paper titled “The perceptron: a probabilistic model for information storage and organization in the brain.” In July that year, the US Office of Naval Research showed off the perceptron. A room-sized IBM 704 computer was fed a series of punch cards, and after 50 trials it taught itself to tell cards marked on the left from cards marked on the right. The New York Times headline read: “New Navy device learns by doing.”

The perceptron, inspired by the way neurons work together in the brain, was a single-layer neural network. In 1969, Marvin Minsky and Seymour Papert attacked Rosenblatt’s work in their book “Perceptrons,” a book said to have essentially sealed its fate. Even so, Cornell professor Thorsten Joachims says Rosenblatt’s algorithm is still fundamental to how deep networks are trained today.

Later, in 1982, John Hopfield published a paper on a network that can recall a whole stored memory from any large enough part of it. In 1985 Hinton and colleagues published a way of training “Boltzmann machines,” and in 1986 Rumelhart, Hinton and Williams published backpropagation.

On October 8, 2024, the Royal Swedish Academy of Sciences announced that the Nobel Prize in Physics would go to John Hopfield of Princeton University in the US and Geoffrey Hinton of the University of Toronto in Canada, “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” The two used tools from physics to develop methods that are the foundation of today’s powerful machine learning.

Hopfield’s network saves patterns, such as pictures, and recreates them. You can think of its nodes as the dots (pixels) of a picture. It uses physics that describes a material’s properties due to atomic “spin,” a property that makes each atom a tiny magnet. The connections between nodes are given values so that the saved pictures have low “energy.” Feed it a distorted or incomplete picture, and it updates the nodes step by step until it reaches the saved picture most like the one it was fed. Think of each saved picture as a dip in bumpy ground: wherever you set a ball on a slope, it rolls into the nearest dip and stops, and a damaged picture likewise settles on a nearby saved one (this comparison is our own).

Hinton used Hopfield’s network as the foundation for a new network, the “Boltzmann machine.” It uses tools from statistical physics, the science of systems built from many similar parts. Trained on examples, it learns to recognize the characteristic features of a type of data, and it can classify pictures or create new examples of the kind of pattern it was trained on. Hinton built on this work and helped start today’s explosive development of machine learning.

Ellen Moons, chair of the Nobel Committee for Physics, said that physics also uses artificial neural networks in a vast range of areas, such as developing new materials with specific properties.