Image generation with diffusion models

In short: A diffusion model creates images by computing an image step by step out of pure noise. In training it learns to remove noise from images.

In more detail: In training a real image is covered with noise in many steps until only noise remains. The network learns the reverse: predicting the noise component from a noisy image.

In detail

1. Forward (training): image + noise of strength t.
2. Learning: the network (U-Net or transformer, “DiT”) predicts the added noise. Loss: MSE.
3. Generating: start with random noise, in 20–50 steps remove some noise each time until an image emerges.
4. Text control: a text encoder (CLIP, T5) supplies vectors the network attends to via cross-attention. Classifier-free guidance strengthens the influence of the prompt.
5. Latent space: Stable Diffusion does not compute in pixels but in a smaller space of an autoencoder (VAE). This saves compute.

• Variants: flow matching, consistency models (few steps), video models.
• Control: negative prompts, seed (same seed = same starting noise), ControlNet (pose, edges), LoRA for styles.
• Counterpart: GANs (generator versus discriminator) were common before and are faster but less stable.
• Legal: copyright of the training images is disputed, see EU AI Act and rules for AI.

See also: Vision transformer and CLIP, Convolutional networks (CNN) for images