What is a diffusion model?

How diffusion models work

A diffusion model is built around two processes that mirror each other. The forward (or diffusion) process takes a real image and gradually corrupts it by adding small amounts of Gaussian noise over many steps, until after enough steps the image is indistinguishable from pure random static. This process is fixed and requires no learning; it simply defines a path from real data to noise.

The reverse process is where the intelligence lives. A neural network is trained to undo one step of noising at a time: given a noisy image, it estimates what was added and moves the image slightly closer to a clean sample. Applied repeatedly, starting from nothing but random noise, this reverse chain gradually sculpts a coherent, novel image. IBM describes diffusion models as working by "adding noise to training data and then reversing the process to generate new data."

The name comes from thermodynamics. The original 2015 paper by Sohl-Dickstein and colleagues, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics," framed generation as reversing a diffusion process, the same mathematics that describes how a drop of ink spreads through water. Running that spreading backwards, from uniform noise to structure, is the generative act.

Training: learning to remove noise

The breakthrough that made diffusion models practical was the 2020 paper by Jonathan Ho, Ajay Jain, and Pieter Abbeel, "Denoising Diffusion Probabilistic Models" (DDPM). Their key simplification was to train the network on a single, concrete task: given a noised image and a number saying how much noise was added, predict that noise. Subtracting the predicted noise recovers a cleaner image.

Training is therefore self-supervised and stable. The model is shown enormous numbers of real images, each corrupted by a random amount of noise, and it learns to predict the corruption. There is no adversarial opponent and no delicate balancing act, which is what distinguishes diffusion training from the notoriously unstable training of generative adversarial networks (GANs).

The workhorse architecture for image diffusion has been the U-Net, a convolutional network with a symmetric down-and-up structure well suited to processing images at multiple scales. Newer systems increasingly replace or augment it with transformer-based backbones (diffusion transformers), which scale better to large models and to video.

From noise to a prompt: text-to-image

A raw diffusion model generates plausible images, but not the specific image you asked for. Text-to-image tools add conditioning: the text prompt is turned into a numerical embedding, often using a text encoder such as CLIP, and that embedding is fed into the denoising network at every step so the emerging image is steered toward the words. The model is trained on image-caption pairs so it learns which visual features correspond to which language.

A second innovation made this cheap enough for the public. Denoising at full image resolution is expensive, so latent diffusion, introduced by Rombach et al. in 2022, first compresses images into a small latent representation with an autoencoder, runs the entire noising and denoising process in that compact space, and only decodes back to pixels at the end. This is the architecture behind Stable Diffusion, and it is the main reason high-quality text-to-image generation can run on a single consumer GPU.

A tuning knob called classifier-free guidance lets users trade off between fidelity to the prompt and diversity of output, pushing the model to adhere more strictly to the text at the cost of some variety.

Where you see diffusion models

Diffusion models power most of the well-known image generators. Stability AI's Stable Diffusion is the openly documented flagship and, being releasable, seeded a large ecosystem of fine-tuned variants and tools. OpenAI's DALL·E 2 was built on diffusion, and Google's Imagen research system used a diffusion pipeline for photorealistic text-to-image generation. Hugging Face's Diffusers library has become a standard open-source toolkit for building and running these pipelines.

The approach has since expanded well beyond still images. Diffusion underpins video generation systems, including OpenAI's Sora, which is described as a diffusion transformer, and it is used for audio and music generation, 3D asset creation, and scientific applications such as molecule and protein-structure design. Wherever the goal is to synthesize complex, high-dimensional data, diffusion has become a default choice.

It is worth being precise about attribution: the internals of some commercial products (for example, Midjourney) are not fully published, so the safest claims are about systems with documented architectures such as Stable Diffusion, DALL·E 2, and Imagen.

Strengths, limitations, and risks

The main strength of diffusion models is a combination of high sample quality, strong diversity, and stable training. Compared with GANs, they are far less prone to "mode collapse," where a generator produces a narrow range of similar outputs, and they cover the variety in the training data more faithfully. This is why, from roughly 2021 onward, diffusion largely replaced GANs as the leading method for image synthesis.

The central weakness is speed. Because generation requires many sequential denoising steps, sampling is computationally heavy and slower than a single-pass generator. A large research effort, faster samplers such as DDIM, and distillation techniques that cut the number of steps, has narrowed this gap substantially, but it remains an active trade-off between quality and latency.

Outputs also still fail in characteristic ways: distorted hands, garbled text within images, and difficulty with precise counts or spatial relationships. And because these models are trained on vast scrapes of internet images, they raise real concerns about copyright, consent, embedded bias, and misuse, including realistic deepfakes, which is why provenance and disclosure are becoming standard expectations for generated media.

The bigger picture

Diffusion models sit alongside transformers as one of the two architectures that defined the current wave of generative AI. Transformers dominate language; diffusion dominates images and, increasingly, video and other continuous data. Many modern systems combine the two, using transformer components for conditioning or as the denoising backbone.

Their rise illustrates a broader pattern in the field: a mathematically elegant idea (reverse a noising process) sat mostly dormant for years until the right training objective and engineering, DDPM in 2020 and latent diffusion in 2022, turned it into a practical, scalable tool almost overnight. Within about two years, text-to-image went from a research curiosity to a mainstream consumer technology.

For anyone trying to understand generative AI, diffusion models are the answer to "how do the image generators actually work." They do not paint; they denoise, repeatedly guessing what a picture should look like as they strip randomness away, guided by the patterns learned from millions of examples.

Frequently asked questions

How is a diffusion model different from a GAN?

Both generate new images, but they work differently. A GAN pits a generator against a discriminator in an adversarial game, which produces images in a single pass but is notoriously unstable to train and prone to mode collapse (limited variety). A diffusion model instead learns to reverse a step-by-step noising process, denoising from random noise into an image over many steps. Diffusion is more stable to train and produces more diverse output, at the cost of slower generation. Since around 2021, diffusion has largely replaced GANs for high-quality image synthesis.

Does Stable Diffusion use a diffusion model?

Yes. Stable Diffusion is a latent diffusion model: it compresses images into a small latent space with an autoencoder, runs the full noising-and-denoising diffusion process in that compact space, and decodes back to pixels only at the end. Working in latent space is why it is efficient enough to run on a single consumer GPU. Text prompts steer the denoising through a text encoder, so the emerging image matches the words.

What is the forward process versus the reverse process in a diffusion model?

The forward (diffusion) process gradually adds Gaussian noise to a real image over many steps until it becomes pure random noise; it is fixed and involves no learning. The reverse process is the learned part: a neural network is trained to undo one step of noise at a time, so that starting from pure noise it can denoise step by step into a coherent, novel image. Generation is the reverse process run from random noise.

Why are diffusion models slow to generate images?

Because generation is sequential: the model must run many denoising steps in order, each a full pass through a neural network, to go from random noise to a finished image. Original methods used hundreds or thousands of steps. Faster samplers such as DDIM and distillation techniques have cut this to a few dozen or even a handful of steps, greatly improving speed, but there is still a trade-off between the number of steps, generation time, and output quality.

Who invented diffusion models?

The core idea was introduced by Jascha Sohl-Dickstein and colleagues in a 2015 paper that framed generation as reversing a diffusion (noising) process borrowed from thermodynamics. The approach became practical for high-quality image generation with the 2020 paper "Denoising Diffusion Probabilistic Models" (DDPM) by Jonathan Ho, Ajay Jain, and Pieter Abbeel, and was made efficient for consumer text-to-image with latent diffusion by Rombach et al. in 2022, the basis of Stable Diffusion.

Can diffusion models generate video and audio, not just images?

Yes. The same denoising principle extends to any high-dimensional data. Diffusion underpins video generators such as OpenAI's Sora (described as a diffusion transformer), as well as audio and music generation, 3D asset creation, and scientific uses like molecule and protein-structure design. Images were simply the first domain where diffusion achieved dramatic, widely visible results.

Related

Explainers: GenAI, Token / tokenization, LLM

Glossary: Generative AI, Transformer (architecture), Text-to-image

Sources

• Ho, Jain & Abbeel (2020) — Denoising Diffusion Probabilistic Models
• Sohl-Dickstein et al. (2015) — Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Rombach et al. (2022) — High-Resolution Image Synthesis with Latent Diffusion Models
• IBM Think — What are diffusion models?
• Hugging Face — Diffusers documentation