How diffusion models paint from noise
Static to masterpiece, step by step. The visual-essay version of our most-watched video.
Type "a lighthouse in a thunderstorm, oil painting" into an image generator and, a few seconds later, a picture appears that has never existed before. Most of today's leading image and video generators - Stable Diffusion, DALL·E 3, Midjourney, Imagen and many video models - rely on the same core idea: diffusion.
The idea sounds backwards at first. To teach a model to create images, you teach it to remove noise from them. This breakdown walks through how that works, step by step.
Step 1: learn to destroy an image
Start with a real photograph. Add a small amount of random noise - the kind of grain you see in a low-light photo. Then add a little more. Repeat this hundreds of times and the image dissolves into pure static, indistinguishable from random noise. This is called the forward process.
The forward process involves no learning at all. It is a fixed recipe, and it is easy to compute: for any image and any step, we know exactly how much noise has been mixed in.
The forward process: an image is gradually buried in noise. No AI is needed for this part.
Step 2: learn to undo it
Now the clever part. We train a neural network on a simple task: given a noisy image and the step number, predict the noise that was added. If the model can tell you which part of the picture is noise, you can subtract it and get a slightly cleaner image.
Training data is effectively unlimited, because we can create noisy examples of any image at any noise level on demand. The network sees millions of images at every stage of corruption and becomes an expert at estimating noise.
Why predict the noise?: Predicting "the noise in this image" turns out to be an easier, more stable learning target than predicting the clean image directly. Removing the predicted noise a little at a time also lets the model refine details gradually rather than all at once.
Step 3: generate by running it backwards
To create a new image, start from pure random noise - no photo at all. Ask the model what noise it sees, remove a portion of it, and repeat. Over many steps (typically 20 to 50 with modern samplers), structure emerges: first rough shapes and colour regions, then edges and objects, and finally fine texture.
Because the starting noise is random, every run produces a different image. Fix the random seed and you get the same starting noise - and the same image - again.
The reverse process: the trained model removes noise step by step. Coarse structure appears first, detail last.
Step 4: steering with text
Removing noise alone would produce random but plausible images. To get your lighthouse, the denoiser must be conditioned on the prompt.
A text encoder turns the prompt into numerical vectors that capture its meaning. Systems have used encoders such as CLIP (trained to match images with captions) and T5 (a large language model).
At each denoising step, the network looks at those text vectors through cross-attention - the same attention mechanism used in transformers, but with image regions asking questions of the prompt's words.
The prediction of "what noise to remove" is therefore nudged toward images that match the description.
Guidance: how literally to follow the prompt
Most systems use a trick called classifier-free guidance. At each step the model makes two predictions - one with the prompt and one without - and pushes the result further in the direction of the prompted version. The guidance scale controls how hard it pushes.
The guidance scale trades creativity for prompt-faithfulness. Typical values sit in the middle.
Step 5: working in latent space
Running hundreds of denoising steps on a full-resolution image is computationally expensive. Latent diffusion, the approach popularised by Stable Diffusion in 2022, solves this by working in a compressed space.
An autoencoder first learns to squeeze images into a much smaller grid of numbers - the latent - and to expand them back. Diffusion happens entirely in that compact latent space, and only at the very end does a decoder turn the final latent into full-resolution pixels. This is a major reason high-quality image generation became possible on consumer graphics cards.
A latent diffusion system. The denoiser runs many times in the small latent space; the decoder runs once.
The same engine, many tools
Once you have a model that removes noise under guidance, many editing features fall out naturally.
Image-to-image: instead of starting from pure noise, partially noise an existing picture and denoise it toward a new prompt. Less noise keeps more of the original.
Inpainting: noise only a masked region - a person you want removed, a sky you want changed - and regenerate it to blend with the rest.
Outpainting: extend the canvas and generate what lies beyond the original borders.
Structural control: add-on networks such as ControlNet condition the process on a sketch, depth map or human pose, so the composition follows your layout.
Video: extend the same idea across time, denoising many frames together so they stay consistent.
Limitations worth knowing
Speed. Many sequential steps make diffusion slower than single-pass methods, though distillation techniques now allow good results in very few steps.
Text and counting. Models historically struggled with legible lettering, exact numbers of objects and precise spatial relationships. Newer systems are much better, but not flawless.
Training data questions. Models learn from very large collections of images scraped from the web, raising ongoing legal and ethical debates about artists' work and consent.
Misuse. Realistic generation makes convincing fakes easier, which is why provenance standards and watermarking efforts are growing.
Summary
Forward process: gradually add noise to real images until only static remains.
Training: teach a network to predict the noise in an image at any stage.
Generation: start from random noise and remove predicted noise step by step.
Conditioning: a text encoder plus cross-attention steers each step toward the prompt; the guidance scale controls how strongly.
Latent diffusion: do the work in a compressed space and decode to pixels at the end, making the whole process far cheaper.