Vision & Multimodal Dec 2021

High-Resolution Image Synthesis with Latent Diffusion Models

Robin Rombach et al. · CVPR 2022

arXiv:2112.10752

In short

Diffusion models working directly on pixels need hundreds of GPU-days to train and are slow to sample. This paper runs diffusion inside the compressed latent space of a pretrained autoencoder instead, which keeps image quality while cutting the cost, and adds cross-attention so text or other inputs can steer what gets generated.

Why it matters

It’s the architecture behind Stable Diffusion, which put high-quality text-to-image generation on ordinary GPUs.

Read first

The 3 Field Guide ideas this paper leans on.

Vision & Multimodal · Landmark Diffusion Model A generative model that learns to denoise data, achieving state-of-the-art image generation (Stable Diffusion, DALL-E 2). Neural Networks · Standard Autoencoder An unsupervised neural network that learns to compress data into a latent representation and reconstruct it, useful for dimensionality reduction. Neural Networks · Standard Latent Space A compressed, learned representation space where similar data points are close together, used in autoencoders and VAEs.

Starting from scratch? The full route 11 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Diffusion Model · read first ✓ understood
A generative model that learns to denoise data, achieving state-of-the-art image generation (Stable Diffusion, DALL-E 2).
Unsupervised Learning ✓ understood
Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.
Autoencoder · read first ✓ understood
An unsupervised neural network that learns to compress data into a latent representation and reconstruct it, useful for dimensionality reduction.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Feature ✓ understood
A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.
Deep Learning ✓ understood
A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.
Representation Learning ✓ understood
Learning useful features or representations of data automatically, rather than hand-crafting them.
Embedding ✓ understood
A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.
Latent Space · read first ✓ understood
A compressed, learned representation space where similar data points are close together, used in autoencoders and VAEs.

Nearby papers

Canon · 2020 Denoising Diffusion Probabilistic Models DDPM is the recipe behind Stable Diffusion, DALL·E, Midjourney and today’s video models. Frontier · #4 Qwen-Image Technical Report Legible, editable text in generated images was a long-standing weakness; this open model largely fixes it. Frontier · #15 Seedream 4.0: Toward Next-generation Multimodal Image Generation It set the bar for commercial image generation and editing in one fast system. Frontier · #16 Diffusion Transformers with Representation Autoencoders It argues the generation and understanding sides of vision should share one latent space.