TL;DR
GANs, VAEs and diffusion models all turn random noise into realistic data, but each trades away something different: speed, sharpness, stability or variety. This guide explains how each one works, compares them side by side and shows where each is used today. Good news for the three engineers arguing over which to pick: modern image systems quietly use all three ideas at once.
Start with these three sections:
Say a product team needs to generate images, and three engineers pitch three models. One wants a GAN because it produces an image in a single pass. One wants a VAE because it trains without drama. One wants a diffusion model because that is what the best image generators use. All three are partly right.
Here is the short answer to diffusion vs GAN vs VAE. GANs give sharp samples and fast generation but are hard to train and can miss parts of the data. VAEs train stably and give a useful compressed representation, but their samples tend to be blurry. Diffusion models give the best quality and variety, at the cost of many steps per sample. Today the three are often combined: Stable Diffusion runs diffusion inside a VAE’s latent space, and its autoencoder is trained with an adversarial loss borrowed from GANs. This guide explains how each works, compares them side by side and shows where each is used now.
Key Takeaways
- A GAN trains a generator against a discriminator. Samples are sharp and take one forward pass, but training can be unstable and suffer mode collapse.
- A VAE learns to compress data into a latent space and decode it back. It trains reliably and covers the data well, but samples are often blurry.
- A diffusion model learns to remove noise step by step. It gives the highest quality and variety, but sampling needs many network calls.
- Most current image and video generators are diffusion or flow-matching models, usually running in a VAE latent space.
- GANs and VAEs did not disappear. GANs power fast audio vocoders and few-step distilled generators, and VAEs serve as the compression stage inside diffusion systems.
What Is a Generative Model?
A generative model is a model that learns the patterns in a dataset well enough to create new samples that look like they came from it. Given thousands of photos of faces, it learns to produce new faces that were never in the training set.
GANs, VAEs and diffusion models are three deep learning answers to the same question: how do you turn random numbers into realistic data? They differ in how they train, how they sample and what they trade away.
How Do GANs Work?
A generative adversarial network (GAN) is a pair of neural networks trained against each other. The generator turns random noise into a sample. The discriminator looks at real samples and generated ones and tries to tell them apart. The generator improves by learning to fool the discriminator, and the discriminator improves by catching it.
Ian Goodfellow and colleagues introduced the idea in the 2014 paper “Generative Adversarial Nets”. The design has two strong points. Sampling is one forward pass through the generator, so it is fast. And because the discriminator punishes anything that looks fake, samples tend to be sharp.
The weak points come from the same contest. Training is a balance between two networks, and if one gets too far ahead, learning stalls or oscillates. The best-known failure is mode collapse: the generator finds a few outputs that fool the discriminator and keeps producing them. A face GAN with mode collapse might generate convincing faces that all share the same age and pose.
How Do VAEs Work?
A variational autoencoder (VAE) is an encoder and a decoder trained together. The encoder maps each input to a distribution in a lower-dimensional latent space, and the decoder maps points from that space back to data. Diederik Kingma and Max Welling introduced it in “Auto-Encoding Variational Bayes” (2013).
Training balances two goals: reconstruct the input well, and keep the latent space close to a simple distribution, usually a standard normal. That second goal is what makes generation possible. Once trained, you sample a random point from the normal distribution and decode it.
VAEs train with ordinary gradient descent on one loss, so they are stable and predictable. They also give you an encoder, which GANs do not, so you can map real data into the latent space, compare samples or interpolate between them. The common complaint is blur. A simple pixel-level reconstruction loss rewards the decoder for predicting an average when it is unsure, and an average of several plausible images looks soft.
How Do Diffusion Models Work?
A diffusion model is a model that learns to reverse a gradual noising process. During training, you take a real image, add a known amount of Gaussian noise, and train a network to predict that noise. To generate, you start from pure noise and apply the network many times, removing a bit of noise at each step until an image remains.
The modern form comes from “Denoising Diffusion Probabilistic Models” (DDPM) by Jonathan Ho, Ajay Jain and Pieter Abbeel (2020). On CIFAR-10 it reported an FID of 3.17, a state-of-the-art score at the time. DDPM used 1,000 noise steps with a linear schedule. This code shows how much of the original image survives at different steps:
import numpy as np
# Linear noise schedule from the DDPM paper (Ho et al., 2020): T = 1000,
# beta rising from 0.0001 to 0.02.
T = 1000
betas = np.linspace(1e-4, 0.02, T)
alpha_bar = np.cumprod(1.0 - betas)
# Closed-form forward process: x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * noise
for t in [1, 100, 250, 500, 1000]:
signal = np.sqrt(alpha_bar[t - 1])
noise = np.sqrt(1.0 - alpha_bar[t - 1])
print(f"step {t:4d}: signal weight {signal:.3f}, noise weight {noise:.3f}") Output:
step 1: signal weight 1.000, noise weight 0.010
step 100: signal weight 0.947, noise weight 0.321
step 250: signal weight 0.724, noise weight 0.690
step 500: signal weight 0.280, noise weight 0.960
step 1000: signal weight 0.006, noise weight 1.000 By step 1,000 almost nothing of the image is left, which is why generation can start from pure noise. It also shows the cost: the original DDPM sampler runs the network once per step, 1,000 times for one image, where a GAN runs once.
Diffusion training is stable for the same reason VAE training is: it is a plain regression loss with no adversary. Each step only has to remove a little noise, which is an easier job than producing a full image at once, and that is a large part of why diffusion models reach high quality and good variety.
Diffusion vs GAN vs VAE: Side-by-Side Comparison
The table sums up the general trade-offs. Specific models can do better or worse on any row, so treat it as a starting point.
| Aspect | GAN | VAE | Diffusion model |
|---|---|---|---|
| How it trains | Generator vs discriminator game | Reconstruction plus a latent-space penalty | Predict the noise added to data |
| Training stability | Hard: sensitive to settings, can collapse | Easy: one stable loss | Easy: one stable regression loss |
| Sample quality | Sharp, high fidelity in narrow domains | Often blurry with simple losses | Highest overall fidelity |
| Variety (mode coverage) | Can miss parts of the data | Good coverage | Good coverage |
| Sampling speed | Fast: one forward pass | Fast: one decoder pass | Slow: tens to hundreds of steps, fewer with distillation |
| Has an encoder | No | Yes | Not by default |
| Typical uses today | Real-time generation, audio vocoders, adversarial losses in other models | Latent spaces for diffusion, tokenizers, scientific data | Text-to-image, video, audio and image editing |
A useful way to read the table: GANs trade stability and variety for speed and sharpness, VAEs trade sharpness for stability and an encoder, and diffusion models trade speed for quality and variety.
Where Is Each Model Used Today?
Diffusion Models Today
Diffusion is the default for text-to-image and video. Stable Diffusion is built on “High-Resolution Image Synthesis with Latent Diffusion Models” by Robin Rombach and colleagues, which runs the diffusion process in a compressed latent space instead of on pixels. Stable Diffusion 3 moved to rectified flow, a close relative of diffusion that learns straighter paths from noise to image, often grouped with diffusion under the name flow matching. OpenAI’s DALL-E 2 and DALL-E 3 are also diffusion-based; our gallery of AI-generated images of Peru was made with DALL-E.
The speed problem has been reduced, not solved. Samplers such as DDIM cut the step count from 1,000 to a few dozen, and distillation methods train a student model to produce good images in one to four steps.
VAEs Today
The VAE’s biggest job today is inside other models. In latent diffusion, a VAE-style autoencoder compresses the image first (Stable Diffusion v1 maps a 512 by 512 image to a 64 by 64 latent with 4 channels), the diffusion model works in that latent space, and the decoder turns the result back into pixels. That is a 48-fold reduction in the number of values the diffusion model handles.
Discrete versions such as VQ-VAE turn images and audio into tokens that a Transformer can model. VAEs also remain common in science and analytics, where the encoder and a structured latent space matter more than photo-quality samples. Our guide to time series forecasting with VAEs shows one such use.
GANs Today
GANs are no longer the default for images, but the adversarial idea is everywhere. The latent diffusion paper trains its autoencoder with a perceptual loss plus a patch-based adversarial objective, which is the GAN trick used to keep reconstructions sharp. GAN-based vocoders such as HiFi-GAN turn spectrograms into audio fast enough for speech systems. Adversarial distillation is one of the methods behind diffusion models that generate in a single step or a handful of steps. And GANs remain a practical choice when you need real-time generation in a narrow domain, such as faces or a single product line.
How to Choose Between Them
- You need the best quality and can afford compute: use a pretrained diffusion or flow-matching model and fine-tune it, rather than training from scratch.
- You need generation in milliseconds: look at distilled few-step diffusion models first, then GANs if your domain is narrow.
- You need a compressed representation, anomaly scores or interpolation: use a VAE.
- You are training on your own model outputs: watch for quality loss over generations, a problem we cover in our article on model collapse.
Back to the Three Engineers
The three engineers from the opening do not have to pick one winner. A typical modern image system uses all three ideas: a VAE to compress images, a diffusion model to generate in that compressed space, and an adversarial loss or distillation step to keep results sharp and fast. The real decision is which pretrained model to start from and how many sampling steps the product can afford.
Frequently Asked Questions
Are Diffusion Models Better Than GANs?
For general text-to-image quality and variety, yes, which is why most current image generators use diffusion or flow matching. GANs still win on raw sampling speed, since they need one forward pass. Distilled diffusion models have narrowed that gap.
Why Are VAE Images Blurry?
With a simple pixel-level reconstruction loss, the decoder is rewarded for predicting an average when several outputs are plausible, and averages look soft. Adding a perceptual or adversarial loss, as latent diffusion autoencoders do, produces much sharper reconstructions.
Does Stable Diffusion Use a VAE?
Yes. Stable Diffusion encodes images into a latent space with a VAE-style autoencoder, runs the diffusion process there and decodes the result back to pixels. Working in the latent space is what makes it practical to train and run at high resolution.
What Is Mode Collapse in GANs?
Mode collapse is when a GAN’s generator produces only a narrow set of outputs that reliably fool the discriminator, instead of covering the full variety of the training data. Samples can look realistic one by one while lacking diversity as a group.




