What is deep learning?

The number of layers. In a 2015 review in the journal Nature, Yann LeCun, Yoshua Bengio and Geoffrey Hinton described deep learning as letting computer models made of many processing layers learn to represent data at multiple levels of abstraction. MIT (Massachusetts Institute of Technology) News likewise explains that the “deep” in deep learning refers to the depth of the network’s layers.

In a neural network, small parts that take in numbers and calculate (neurons) are lined up to form a layer. When data goes into the first layer, the input layer, each layer takes the results from the one before, calculates, and passes them on, until the last layer, the output layer, gives the answer. Google’s machine learning glossary calls a neural network with more than one layer between input and output (more than one hidden layer) a “deep model.” How a neural network works inside (nodes, weights and how it learns) is explained on the “Neural network” page.

According to the MIT News explainer, networks had one layer in the 1960s and two or three in the 1980s, but modern GPUs let them grow to 10, 15, even 50 layers (as of 2017). In the ILSVRC photo contest, the 2014 winner, GoogLeNet, had 22 layers, and the 2015 winner, ResNet, had up to 152.

Each layer can combine what the layer before it found to represent something more complex. In 2015, Google researchers explained that a network that recognizes photos usually has 10 to 30 stacked layers: the first layer may look for edges or corners, middle layers combine those to look for overall shapes or parts, like a door or a leaf, and the last layers put them together into a complete interpretation.

In 2013, Matthew Zeiler and Rob Fergus made it possible to see what kinds of images each layer of a trained network responds to most strongly. The second layer responded to corners and combinations of edges and colors; the third to textures like mesh patterns and text; the fourth to parts like dog faces and birds’ legs; and the fifth to whole objects like dogs and keyboards. The paper says this shows the features are built up in levels.

The key point is that no person decides what each layer picks up. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams wrote in Nature about adjusting the strengths of the connections again and again to shrink the gap between the network’s answer and the right one (backpropagation). When you do, the in-between neurons, neither input nor output, come to represent features that matter for the task. The 2015 ResNet paper also says the levels of features can be enriched by the number of stacked layers (depth).

Deep learning is a kind of machine learning too, but it gets its clues (features) differently. In earlier methods, people often designed the recipe for turning a photo into features, deciding what to pay attention to. For example, the NEC team that won the ILSVRC photo contest in 2010 turned photos into lists of numbers using feature recipes designed by researchers, called SIFT and LBP, and then sorted them with machine learning.

The University of Toronto team that won in 2012 fed the color values of a photo’s pixels (RGB values) straight into a deep network and trained it (according to the paper summing up the contest). Google’s machine learning course likewise explains that a neural network learns good ways to combine the clues in the data on its own as it trains, so people don’t have to try out many options by hand.

Some networks have even found clues without being told the answers. In 2012, a Google research team showed a network lots of unlabeled images taken from YouTube videos. No one ever told it “this is a cat,” yet one neuron inside the network came to respond to cats. According to the paper, the network had 9 layers and about 1 billion connections, and it was trained for three days on 1,000 computers (16,000 processor cores).

How deep learning relates to AI and machine learning is on the “Machine learning” page: machine learning is inside AI, and deep learning is inside machine learning, like nested boxes.

The idea itself had been around for a long time. According to the MIT News explainer, neural networks were first proposed in the 1940s, became a busy research area again in the 1980s, fell out of favor in the 2000s, and came roaring back in the 2010s.

The same explainer says the comeback was fueled largely by computing power. Chips built for drawing game graphics (GPUs) pack thousands of simple processing cores onto one chip, and researchers realized that this design is remarkably like a neural network. A Computer History Museum article likewise explains that training a neural network involves lots of repeated multiplications, best done at the same time, and that NVIDIA’s CUDA system, released in 2007, made GPUs easier to use for work other than graphics.

The data came together too. ImageNet, a collection of photos that the researcher Fei-Fei Li began in 2006, was completed in 2009 and was larger than any earlier image collection by several orders of magnitude (the museum article). By August 2014 it held about 14.2 million labeled photos. The photo contest uses about 1.2 million of them for training, sorted into 1,000 categories (the paper summing up the contest).

Better ways of training were added as well. In 2012, Hinton and colleagues reported that “dropout,” randomly switching off half of the neurons for each training example, greatly reduces overfitting. The museum article says the University of Toronto’s 2012 network (AlexNet) was the first to bring deep neural networks, big datasets and GPU computing together, and that each of these needed the others.

ILSVRC, a contest to name what is in photos, was held every year from 2010. For each photo, a system may give up to five guesses, and the contest compares the share of photos where none of the five was right (lower is better). When the contest began, the winner’s rate was 28.2% (the paper summing up the contest).

In 2012, a team (SuperVision) of University of Toronto graduate students Alex Krizhevsky and Ilya Sutskever and their advisor Hinton got 16.4% with a deep network that had 60 million parameters (the numbers used in its calculations); a version also trained in advance on other photos got 15.3%. Second place got 26.2%, a wide gap. The network was trained on a computer with two NVIDIA GPUs, in Krizhevsky’s bedroom at his parents’ house (the museum article).

The paper summing up the contest calls this a turning point, when large deep neural networks entered the scene. The museum article says that before AlexNet, almost none of the leading computer vision papers used neural networks, and afterward almost all of them did. By 2014, all the top teams in the contest relied heavily on deep learning (convolutional neural networks). The winning error rate fell to 6.7% in 2014 and 3.57% in 2015.

How about compared with people? In the contest paper, one expert who had practiced on 500 photos was tested on 1,500 photos and got 5.1%. On the same photos, the 2014 winner, GoogLeNet, got 6.8%, so at that point the person was better. In February 2015, Microsoft researchers got 4.94% and wrote that, to their knowledge, it was the first result to surpass human-level performance (5.1%). But the kinds of mistakes differ: according to the contest paper, the machine (GoogLeNet) struggles with very small or thin objects, while the person tends to slip when telling apart closely similar fine-grained categories.

Not necessarily. In 2015, Microsoft researchers asked in the ResNet paper, “Is learning better networks as easy as stacking more layers?” When they simply stacked layers and tested the networks on a set of photos (CIFAR-10), a 56-layer network made more mistakes than a 20-layer one, both on the data it learned from and on test data.

The paper says this was not caused by overfitting. Overfitting means memorizing the training data so well that the network does well on it but fails on data it has never seen. With overfitting, the mistakes on the data used for learning would, if anything, be small (see the “Machine learning” page). The deeper the network, the harder it becomes to train it well in the first place.

So the paper added “shortcuts” that skip over a few layers, passing the input straight ahead and adding it to what the layers produce. The shortcuts add no extra parameters and no extra calculation. With them, even 152 layers could be trained, and a combination of ResNets of different depths won first place in classification at the 2015 ILSVRC photo contest (3.57%). Depth makes a network stronger only once training is done the right way.

It can be fooled by changes too small for people to notice. In 2013, Christian Szegedy and colleagues showed that adding a change to a photo so small that the human eye can’t see it can make a network misidentify the photo. The same change could also fool a different network trained on a different subset of the same data.

Even the people who build these networks don’t always know why they work. In 2015, Google researchers wrote that they actually understand surprisingly little of why certain models work and others don’t. Research that looks at what each layer picks up, like Zeiler and Fergus’s, is partly a way to find out.

Needing lots of data and computing is another weak spot. The Google experiment in which a cat-detecting neuron emerged took three days on 1,000 computers. And because deep learning is a kind of machine learning, it also picks up any skew in the data it learns from (see the “Machine learning” page).

Beyond photos, it also succeeded early in speech recognition, which turns voices into text. A 2012 paper written jointly by four research groups reported that deep networks with many hidden layers, trained with new methods, beat the method used until then (Gaussian mixture models) on a variety of tests, sometimes by a large margin.

AIs that write text, like ChatGPT (large language models), are deep learning too. Their most widely used design, the Transformer, stacks many layers into an enormous neural network (Google’s course). The 2020 GPT-3 had 96 layers and 175 billion parameters (see the “Transformer” and “Large language model” pages).

The 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” The Royal Swedish Academy of Sciences, which announced it, wrote that Hinton invented a method that can find properties in data on its own, and that he helped start today’s explosive development of machine learning.