Diffusion Models in AI Image Generation – Part 1: Foundations

Introduction: The Revolution in AI Image Generation

When we think of AI image generation today, we immediately picture models like DALL·E, Midjourney, or Stable Diffusion—technologies capable of producing astonishingly detailed artwork from simple text descriptions. This remarkable capability, which seemed like science fiction just a few years ago, is largely powered by a family of algorithms known as diffusion models.

At their core, denoising diffusion probabilistic models (DDPMs) work by learning to gradually remove noise from an image, transforming random patterns into coherent visual content. This innovative approach has revolutionized generative AI, enabling unprecedented quality and control in image synthesis.

In this two-part series, we’ll explore:

Part 1: Foundations – The challenges of pixel-by-pixel generation and why researchers sought alternatives – The fundamental diffusion concept: adding and removing noise – Why training models to predict noise offers superior results – The mathematics behind the diffusion process

Part 2: Applications & Advancements – Text-to-image generation techniques – Accelerating sampling with advanced methods like Latent Consistency Models – Practical implementation and code examples – The future of diffusion-based generative AI

By the end of this series, you’ll understand both the theoretical underpinnings and practical applications of diffusion models—equipping you with knowledge of why they’ve become the dominant paradigm in modern AI image generation.

The Challenge: How to Generate Images

Before diving into diffusion models, it’s important to understand the fundamental challenge of image generation: how can we teach an AI to create images that look realistic, detailed, and coherent?

The Autoregressive Approach (and Its Limitations)

One intuitive approach to image generation is treating it as a pixel-by-pixel prediction task, similar to how language models predict the next word in a sequence. This strategy is called autoregressive modeling.

How Autoregressive Generation Works:
1. Start with an empty canvas
2. Predict the first pixel (usually top-left)
3. Move to the next position (often left-to-right, top-to-bottom)
4. Use all previously generated pixels as context to predict the next one
5. Repeat until the entire image is filled

Advantages:
– Conceptually straightforward—each step is a simple prediction
– The approach has proven highly effective in language modeling (e.g., GPT models)
– Can model complex distributions with enough computational power

Critical Limitations for Images:
1. Extreme Inefficiency: A 1024×1024 pixel image requires over a million sequential predictions
2. Slow Generation: Even with optimizations like predicting patches instead of individual pixels, the process remains prohibitively slow
3. Local Dependencies: Pixel values are strongly correlated with their neighbors, creating complex dependencies
4. Difficulty Capturing Global Structure: The step-by-step approach can struggle with maintaining coherent, large-scale features

While autoregressive image models do exist and can produce impressive results (like PixelCNN), the sequential generation process makes them impractical for many real-world applications where speed and scalability matter.

The Diffusion Revolution: Learning to Denoise

Diffusion models take a fundamentally different approach. Instead of directly predicting pixel values, they ask:

What if we gradually add noise to images and then train a model to reverse this process—progressively removing noise to recover clean images?

This insight leads to a two-phase process that forms the backbone of all diffusion models.

The Forward Process: Adding Noise Systematically

The forward process is deterministic and involves gradually corrupting an image with noise:

1. Start with a clean image from your training dataset
2. Apply noise incrementally through a series of steps
3. Generate increasingly noisy versions
4. At the final step , the image becomes indistinguishable from pure random noise

Mathematically, each step of the forward process can be described as:

Where: – controls how much of the original signal remains – is random Gaussian noise – The sequence forms a noise schedule that determines how quickly the image dissolves into noise

This forward process is purely for training and establishing the mathematical framework—we never use it during actual image generation.

The Reverse Process: Learning to Denoise

The true power of diffusion models lies in the reverse process, where a neural network learns to progressively denoise an image:

1. Start with random noise
2. Apply the neural network to estimate a slightly less noisy version
3. Repeat the denoising step to get and so on
4. Continue until reaching —a completely new, synthesized image

During training, the neural network observes examples of noisy images at various corruption levels along with their corresponding less-noisy versions (or the original clean images). It learns to reverse the noise addition process step by step.

The Key Insight: Predicting Noise Instead of Images

A critical advancement in diffusion models came with the realization that training the network to predict the noise itself—rather than the denoised image—leads to more stable training and better results.

Why Noise Prediction Works Better

When an image is heavily corrupted with noise (as in the early stages of the reverse process), there’s almost no signal about the original content. If the model tries to directly predict “what the clean image should look like,” it has little information to work with.

In contrast, predicting “what noise was added” turns out to be a more tractable problem:

1. Signal Isolation: The model learns to differentiate between actual content and the random noise
2. Stable Gradients: Especially in early denoising steps when the image is mostly noise, this approach provides clearer learning signals
3. Better Convergence: Training tends to be more stable and converge faster

The Mathematics of Noise Prediction

If our diffusion model is trained to predict the noise component, we can estimate the original image from any noisy state with:

This formula effectively says: “If I know how much noise was added at this step, I can subtract it out (with appropriate scaling) to estimate the original clean image.”

During training, we optimize the model using a simple objective—minimize the difference between the actual noise added and the noise predicted by our model:

Where: – is randomly sampled from the possible timesteps
– is a clean training image
– is the actual noise that was added
– is our model’s prediction of that noise

Visualizing the Process

To understand this intuitively:

1. Early in denoising (e.g., going from to ): The model identifies broad patterns of noise against an almost entirely random background
2. Middle stages (e.g., ): The model distinguishes emerging structures from the diminishing noise
3. Final steps (approaching ): The model makes fine adjustments, removing subtle noise artifacts to reveal detailed features

Code Example: Simplified Diffusion Training Loop

Below is a simplified PyTorch implementation showing the core components of training a diffusion model to predict noise:

import torch
import torch.nn as nn
import torch.nn.functional as F

class SimpleDiffusionModel(nn.Module):
    def __init__(self):
        super().__init__()
        # A simple U-Net-like architecture for demonstration
        # In practice, much more sophisticated models are used
        self.net = nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1),  # Input: Image with 3 channels (RGB)
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, padding=1), # Feature extraction
            nn.ReLU(),
            nn.Conv2d(64, 3, 3, padding=1)   # Output: Predicted noise with 3 channels
        )
        
    def forward(self, x, t):
        # In practice, the timestep t would be encoded and provided to the network
        # This is a simplification - real models encode t and inject it into multiple layers
        return self.net(x)

# Training loop
def train_diffusion(model, dataloader, optimizer, num_timesteps=1000, num_epochs=100):
    # Create a noise schedule (typically linear, cosine, etc.)
    betas = torch.linspace(0.0001, 0.02, num_timesteps)  # Start with small noise, gradually increase
    alphas = 1 - betas
    alphas_cumprod = torch.cumprod(alphas, dim=0)  # Cumulative product of alphas
    
    for epoch in range(num_epochs):
        for batch in dataloader:
            optimizer.zero_grad()
            
            # Get clean images from the dataset
            x_0 = batch
            
            # Randomly sample timesteps - this lets us train on all noise levels simultaneously
            t = torch.randint(0, num_timesteps, (x_0.size(0),))
            
            # Get the corresponding alpha values for these timesteps
            alpha_t = alphas_cumprod[t]
            
            # Reshape alpha_t for proper broadcasting
            alpha_t = alpha_t.view(-1, 1, 1, 1)
            
            # Sample noise
            epsilon = torch.randn_like(x_0)
            
            # Create the noisy image at timestep t
            # This is the forward process equation: x_t = √(αₜ)x₀ + √(1-αₜ)ε
            x_t = torch.sqrt(alpha_t) * x_0 + torch.sqrt(1 - alpha_t) * epsilon
            
            # Predict the noise using our model
            predicted_noise = model(x_t, t)
            
            # Calculate the loss (mean squared error between actual and predicted noise)
            loss = F.mse_loss(predicted_noise, epsilon)
            
            # Backpropagate and update model
            loss.backward()
            optimizer.step()
            
        print(f"Epoch {epoch}, Loss: {loss.item()}")

This simplified code illustrates the core components of diffusion model training:
1. Creating a noise schedule
2. Sampling random timesteps
3. Adding noise to clean images according to those timesteps
4. Training the model to predict that noise

In practice, diffusion models use more sophisticated architectures with additional features like attention mechanisms, timestep embeddings, and specialized normalization techniques.

Sampling Efficiency: The Computational Challenge

A significant practical concern with early diffusion models was sampling speed. The original formulations required hundreds or even thousands of denoising steps to produce high-quality images, making them too slow for many applications.

The Sampling Speed Trade-off

The table below illustrates the trade-off between sampling steps and generation quality:

Several techniques have emerged to address this challenge:

1. Improved Samplers: Algorithms like DDIM (Denoising Diffusion Implicit Models) and DPM-Solver allow for much fewer steps than the original DDPM formulation
2. Better Noise Schedules: Carefully designed schedules (linear, cosine, etc.) can make the denoising process more efficient
3. Model Architecture Improvements: Enhanced neural network designs that better capture the relationship between noise and image content

These advances have collectively reduced the typical number of required steps from thousands to mere dozens, making diffusion models viable for real-world applications.

Code Example: Basic Sampling with DDIM

Here’s a simplified implementation of the DDIM sampling procedure, which offers better efficiency than the original DDPM approach:

def sample_ddim(model, shape, num_timesteps=1000, num_steps=50):
    """
    Sample an image using DDIM (deterministic) sampling.
    
    Args:
        model: The trained diffusion model
        shape: Shape of the image to generate (batch_size, channels, height, width)
        num_timesteps: The total number of steps in the original diffusion process
        num_steps: Number of sampling steps to use (fewer = faster but potentially lower quality)
    
    Returns:
        The generated image
    """
    device = next(model.parameters()).device
    
    # Start with random noise
    x_t = torch.randn(shape, device=device)
    
    # Create a reversed sampling schedule - we're going backwards from T to 0
    # Fewer steps than training for efficiency
    timesteps = torch.linspace(num_timesteps - 1, 0, num_steps + 1).long()
    
    for i in range(num_steps):
        # Current and next timestep
        t = timesteps[i]
        t_next = timesteps[i + 1]
        
        # Calculate alphas
        alpha_t = alphas_cumprod[t]
        alpha_t_next = alphas_cumprod[t_next] if t_next >= 0 else torch.tensor(1.0)
        
        # Predict noise
        predicted_noise = model(x_t, t)
        
        # Estimate the clean image
        # This uses our noise prediction to estimate what x₀ might be
        x_0_pred = (x_t - torch.sqrt(1 - alpha_t) * predicted_noise) / torch.sqrt(alpha_t)
        
        # DDIM deterministic update - different from the stochastic DDPM update
        direction = torch.sqrt(1 - alpha_t_next) * predicted_noise
        x_t = torch.sqrt(alpha_t_next) * x_0_pred + direction
    
    return x_t  # Final denoised image

This sampling procedure enables much faster generation by efficiently traversing the timesteps, making deterministic (rather than stochastic) predictions at each step.

Conclusion: The Foundation of Modern AI Image Generation

In Part 1 of our series, we’ve covered the fundamental concepts that make diffusion models work:

• The limitations of pixel-by-pixel autoregressive generation that motivated new approaches
• The forward process of systematically adding noise to images
• The reverse process where neural networks learn to denoise step by step
• The critical insight that predicting noise rather than clean images leads to better results
• The challenges of sampling efficiency and early solutions

These foundations have established diffusion models as the dominant paradigm in AI image generation today. Their ability to create remarkably detailed, coherent images has transformed what’s possible in computational creativity.

In Part 2 on Applications & Advancements, we’ll explore how these fundamental concepts extend to real-world applications like text-to-image generation, cutting-edge techniques like Latent Consistency Models that dramatically improve sampling speed, and practical implementations that put these powerful tools in the hands of developers and artists.

Stay tuned to learn how these theoretical foundations translate into the remarkable AI art generators that are changing creative workflows across industries.