A diffusion model builds an image, video clip, or sound by starting with a field of pure random noise and running it through a trained neural network dozens of times, subtracting a little more of that noise at each pass, until what’s left resembles a coherent picture that matches whatever text prompt guided it. It’s the mechanism behind Stable Diffusion, Midjourney, DALL-E, and OpenAI’s Sora — different products built by different companies, but the same underlying technique for turning words into pixels, and one of the core tools of today’s generative AI boom.
Training: teaching a network to recognize noise
A diffusion model doesn’t learn by memorizing finished pictures. It learns by destruction. Researchers take a large set of real photos and, over many small steps, add a measured dose of random Gaussian noise to each one, until the original image is buried under static and is statistically indistinguishable from pure noise. They then train a neural network on the reverse question: shown any one of those in-between, partly-noisy images, can it predict exactly which noise was added? A network that gets good at that one narrow task — the approach described in the 2020 Denoising Diffusion Probabilistic Model paper — has, without ever being told what a “cat” or a “cathedral” looks like, learned enough about the structure of real images to undo the damage.
Generation: running the process backward
Making a new image means running that trained network in reverse. It starts from a canvas of random noise — mathematically no different from the noise added during training — and asks the network to predict what noise is present. It subtracts a fraction of that prediction, leaving a slightly cleaner canvas, and repeats. Early diffusion models needed around 1,000 of these passes to produce one image, which is why the first versions were painfully slow; modern samplers reorganize the math to reach a comparable result in as few as 20–50 steps, which is a big part of why today’s AI image generators return a picture in seconds rather than minutes.
How the prompt actually steers the picture
Left alone, this process just produces a plausible-looking but random image — it has to be pointed toward a specific scene. That’s the job of conditioning: a separate text encoder turns your prompt into a numerical representation that’s fed into the denoising network at every single step, so that of all the ways it could remove noise, it favors the ones that drift toward matching your words. A technique called classifier-free guidance controls how tightly the output sticks to the prompt versus how much visual freedom the model keeps — pushed too high, images get harsh and over-saturated; too low, and the prompt gets largely ignored. Our guide to using an AI image generator walks through what that looks like in practice.
Doing it in a smaller space
Running this noise-and-denoise cycle directly on every pixel of a large image is computationally brutal. Latent diffusion — the approach a Munich research group published in 2022, which Stability AI built into the original Stable Diffusion — instead compresses the image into a much smaller mathematical representation first, runs the entire process in that compressed space, and only expands the result back to a full-size image at the very end. That single change is a big reason Stable Diffusion can run on a home GPU instead of a data center.
Beyond still pictures
The same mechanism generalizes past static images. OpenAI has described Sora, its model for AI-generated video, as a diffusion system that denoises video broken into space-and-time “patches” rather than flat 2D pixels — and, notably, swaps in a transformer, the same architecture behind large language models, in place of the image-focused network diffusion systems traditionally use. Diffusion techniques are also used to synthesize audio and, outside media entirely, to generate candidate molecule and protein structures in drug research. Diffusion overtook an older generative approach, the generative adversarial network (GAN), as the dominant method over the past several years mainly because it trains more stably and follows prompts more faithfully — even though it’s heavier to compute per image than a GAN’s single forward pass.
Sources: “Denoising Diffusion Probabilistic Models” (Ho, Jain, Abbeel, 2020); “High-Resolution Image Synthesis with Latent Diffusion Models” (Rombach et al., 2022); OpenAI, “Video generation models as world simulators.”