What is a diffusion model (how AI makes pictures)?

It makes its own practice problems, with answers it can check. Inside a computer, a picture is a list of numbers giving the strength of the colors at each dot, and noise is a list of random numbers of the same length. That is why they can be added and subtracted. It picks a training picture, decides at random how much noise to mix in, and adds that noise. The AI looks at the noisy picture and guesses the noise that was mixed in. Then the numbers inside the AI (its parameters) are nudged a little so that the gap between the guessed noise and the noise actually added gets smaller. This is repeated many times over many pictures. This way of training was shown in a 2020 paper (Ho et al., DDPM), and Stable Diffusion also learns from the difference between the noise added and the noise the AI guesses.

The right answer is known, because it is the noise that was added. Learning by deliberately adding noise and having a model remove it (denoising) is known as a way to learn even from data that no one has labeled, picture by picture, with answers (such as what it shows). To build an AI you can ask with words, it is trained on pairs of pictures and their descriptions.

The AI that guesses the noise is a deep-learning neural network (an AI design that stacks many layers of calculation). DDPM and Stable Diffusion used a design called U-Net, which squeezes a picture down and then brings it back up to full size. A 2022 paper (Peebles and Xie) swapped it for a transformer (the design behind chat AIs) and, on a test of making pictures of a chosen kind (using ImageNet, a large collection of photos of many kinds), got better results than earlier diffusion models.

Because when each step changes things only a tiny bit, the state one step earlier is almost the same as the state now, so it is easy to guess. So the AI only has to learn to guess “the state one step earlier, with just a little less noise.” Mathematics has shown that when each step adds only a tiny bit of noise, undoing that step can be described by the same kind of randomness as adding it (a Gaussian distribution) (Feller’s work in 1949). The 2015 paper that first proposed diffusion models (Sohl-Dickstein et al.) built on this.

So when it starts from pure noise, what does the AI go by? It is not picturing one particular image. Having practiced on many pictures, it has learned a calculation that tells “the parts that don’t look like a real picture are noise.” Each time it subtracts noise with that calculation, the picture moves a little closer to likely shapes, such as mountains or faces. Different starting noise gives a different picture.

DDPM in 2020 split the adding of noise into 1,000 steps, and when making a picture it removes noise 1,000 times, starting from pure noise. Watching along the way, the rough shapes appear first and the fine details come last.

The downside is that making a picture takes time. So ways to cut the number of steps followed one after another. DDIM in 2020 (Song et al.) kept the same training but made pictures 10 to 50 times faster. Stable Diffusion’s released program makes a picture in 50 steps by default.

Because the words go into the calculation that removes the noise. An AI can only calculate with numbers, so in Stable Diffusion, a part that turns text into a list of numbers (a text encoder, taken from another AI called CLIP, which learned from pairs of pictures and captions) first converts your request into numbers. The AI that guesses the noise refers to those numbers at every step. Since it was also trained on pairs of pictures and descriptions, it can remove the noise so that the picture moves toward one that fits the words.

There is also a trick to make it follow the words more strongly. In the method from a 2022 paper (Ho and Salimans, Google), one AI practices guessing the noise both with and without the request (in the paper’s tests, the request was not words but the name of a kind of picture). When making a picture, at every step it guesses the noise both with and without the request, and takes the difference between the two answers (the direction the request adds), multiplied several times over. The bigger the multiplier, the more clearly the picture fits the request, but the less the pictures vary. The released Stable Diffusion (v1-4), too, was trained with the words left out 10% of the time, so that it could also give the without-request answer and this method would work well with words.

DALL·E 2 (OpenAI, 2022) first turns the words into a list of numbers that describes the features of a picture, then uses a diffusion model to turn that into a picture.

It shrinks the picture into a small set of numbers first, and adds and removes noise there (a latent diffusion model). A paper released in December 2021 (Rombach et al.) wrote that training diffusion models directly on a picture’s dots (pixels) can take hundreds of days of GPU computing for the most powerful ones (GPUs are the chips used for AI computing), and that making 50,000 pictures takes about 5 days on one GPU.

In Stable Diffusion, a separately trained AI shrinks a picture of 512×512 dots (3 numbers per dot, for the strength of red, green and blue; 786,432 numbers in all) into four 64×64 layers, 1/8 the size on each side (16,384 numbers). That is about 1/48 as many numbers. After the noise has been removed, the expanding AI paired with the shrinking one turns it back into a full-size picture. The paper says this cut the computing greatly while keeping the fine look of pictures.

In August 2022, Stable Diffusion was released for anyone to use, along with its numbers (parameters). The U-Net that guesses the noise has 860 million numbers and the part that turns words into numbers has 123 million, and it was described as running on a GPU with at least 10 GB of memory.

The idea came in 2015. Researchers at Stanford University and elsewhere (Sohl-Dickstein et al.) took a hint from the physics of things spreading out and mixing, like ink in water (non-equilibrium statistical physics), and proposed training a model on a “diffusion” process that slowly breaks down the structure of data, and on a process that restores it. This is where the name “diffusion” comes from.

In 2019, Song and Ermon proposed a method that learns by adding noise of many strengths to data and makes samples by moving step by step from strong noise to weak. In 2020, DDPM (Ho et al., University of California, Berkeley) showed that a diffusion model could make pictures about as good as a leading GAN (an older kind of picture-making AI, explained below). It also showed that, though they look different, it and the Song and Ermon approach belong to the same family once the equations are rewritten.

In 2021, Dhariwal and Nichol of OpenAI reported, in a paper titled “Diffusion Models Beat GANs on Image Synthesis,” that diffusion models had overtaken GANs, which until then led in picture quality. On a test of making photos of a chosen kind (256×256 dots) with ImageNet, a large collection of photos of many kinds, a measure of how far the set of pictures made is from the set of real photos (FID; lower is closer to real) was 4.59, compared with 6.95 for a leading GAN (BigGAN-deep). A 2023 paper (Carlini et al.) says that over the previous two years or so, diffusion models largely replaced GANs and became the basis of well-known image AIs such as Stable Diffusion, DALL·E 2, Midjourney and Imagen.

The difference from GANs is how they make pictures. A GAN trains an AI that makes pictures by pitting it against an AI that tells real from fake, and it makes a picture in a single pass (more on the “Generative AI” page). A diffusion model removes noise over many steps, so, as the same 2021 paper notes, it is slower than a GAN at making pictures.

Yes. An April 2022 paper (Ho et al.) extended the design used for pictures, in a natural way, into a diffusion model that makes videos. It also showed the first results on a large test of making videos from text.

In February 2024, OpenAI released a technical report on its video AI, Sora. According to the report, Sora is a diffusion model, trained to predict the original clean patches from small noisy patches. It also says Sora is a “diffusion transformer,” which cuts video into “patches,” each a few frames long and a small square of the screen, and handles them with a transformer. OpenAI says it can make up to a minute of high-definition video, but also admits that it does not accurately model the physics of many basic interactions, such as glass shattering. Details of the model are not included in the report.