Deep Generative Models — Course Materials (DGM)

This repository collects lecture slides, assignments (CAs), code notebooks, reports, and reference papers used in the "Deep Generative Models" course (University of Tehran). The materials are organized to be reproducible and educational: each assignment contains an annotated Jupyter notebook, supporting code, and a report.

Course Overview

The "Deep Generative Models" (DGM) course covers advanced topics in machine learning focused on generative modeling techniques. Generative models learn the underlying distribution of data to generate new samples, enabling applications in image synthesis, anomaly detection, data augmentation, and more.

Key topics covered in the course include:

• Variational Autoencoders (VAEs): Probabilistic latent variable models for learning compressed representations and generating new data.
• Normalizing Flows: Invertible transformations that allow exact density estimation and sampling.
• Generative Adversarial Networks (GANs): Adversarial training frameworks for high-quality sample generation.
• Diffusion Models: Denoising diffusion probabilistic models for state-of-the-art image generation.
• Score-based Generative Models: Methods using score functions for sampling from complex distributions.

The course assignments (CA1-CA4) progressively build skills in implementing and evaluating these models on real datasets like CelebA, FashionMNIST, and custom image datasets.

Key Concepts in Deep Generative Models

This section provides a high-level overview of the core mathematical and conceptual foundations that unify the different generative modeling approaches covered in the course.

Probabilistic Generative Modeling

Generative models aim to learn the underlying data distribution $p(\mathbf{x})$ from samples $\mathbf{x} \sim p_{\text{data}}$. The goal is to:

1. Density Estimation: Approximate $p(\mathbf{x})$ or learn a tractable distribution $p_\theta(\mathbf{x})$ that matches the data.
2. Sampling: Generate new samples $\mathbf{x}' \sim p_\theta(\mathbf{x})$ from the learned distribution.
3. Inference: Compute posterior probabilities or latent representations for downstream tasks.

Maximum Likelihood Estimation

Most generative models are trained by maximizing the log-likelihood:

$$ \theta^* = \arg\max_\theta \mathbb{E}{\mathbf{x} \sim p{\text{data}}} [\log p_\theta(\mathbf{x})] $$

This is equivalent to minimizing the KL divergence between data and model distributions:

$$ \theta^* = \arg\min_\theta \text{KL}(p_{\text{data}} || p_\theta) $$

Latent Variable Models

Many generative models introduce latent variables $\mathbf{z}$ to simplify modeling:

• Joint Distribution: $p_\theta(\mathbf{x}, \mathbf{z}) = p_\theta(\mathbf{x}|\mathbf{z}) p(\mathbf{z})$
• Marginal Likelihood: $p_\theta(\mathbf{x}) = \int p_\theta(\mathbf{x}|\mathbf{z}) p(\mathbf{z}) d\mathbf{z}$
• Posterior Inference: $p_\theta(\mathbf{z}|\mathbf{x}) = \frac{p_\theta(\mathbf{x}|\mathbf{z}) p(\mathbf{z})}{p_\theta(\mathbf{x})}$

Amortized Variational Inference

Exact inference in latent models is often intractable. Variational inference approximates posteriors using a recognition model:

• Variational Distribution: $q_\phi(\mathbf{z}|\mathbf{x}) \approx p_\theta(\mathbf{z}|\mathbf{x})$
• Evidence Lower Bound (ELBO): $\log p_\theta(\mathbf{x}) \geq \mathbb{E}{q\phi(\mathbf{z}|\mathbf{x})} [\log p_\theta(\mathbf{x}|\mathbf{z})] - \text{KL}(q_\phi(\mathbf{z}|\mathbf{x}) || p(\mathbf{z}))$

Change of Variables and Normalizing Flows

Normalizing flows provide exact density estimation through invertible transformations:

• Transformation: $\mathbf{z} = f(\mathbf{x})$ where $f$ is invertible
• Density: $p_\mathbf{z}(\mathbf{z}) = p_\mathbf{x}(\mathbf{x}) |\det \frac{\partial f}{\partial \mathbf{x}}|$
• Composition: Complex transformations built from simple invertible layers

Adversarial Training

GANs use adversarial objectives instead of explicit likelihoods:

• Generator: $G: \mathbf{z} \mapsto \mathbf{x}$, learns to fool discriminator
• Discriminator: $D: \mathbf{x} \mapsto [0,1]$, learns to distinguish real from fake
• Minimax Objective: $\min_G \max_D \mathbb{E}{\mathbf{x}} [\log D(\mathbf{x})] + \mathbb{E}{\mathbf{z}} [\log (1 - D(G(\mathbf{z})))]$

Diffusion Processes

Diffusion models gradually add noise and learn to reverse the process:

• Forward Process: $q(\mathbf{x}t | \mathbf{x}{t-1}) = \mathcal{N}(\mathbf{x}t; \sqrt{1-\beta_t} \mathbf{x}{t-1}, \beta_t \mathbf{I})$
• Reverse Process: $p_\theta(\mathbf{x}{t-1} | \mathbf{x}t) = \mathcal{N}(\mathbf{x}{t-1}; \boldsymbol{\mu}\theta(\mathbf{x}t, t), \boldsymbol{\Sigma}\theta(\mathbf{x}_t, t))$
• Training: Predict noise added at each timestep

Score-Based Generative Modeling

Score-based models learn the score function (gradient of log-density):

• Score Function: $\nabla_\mathbf{x} \log p_t(\mathbf{x})$ where $p_t$ is a noisy version of data
• Score Matching: Minimize $\mathbb{E}{p_t(\mathbf{x})} [\frac{1}{2} || \mathbf{s}\theta(\mathbf{x}, t) - \nabla_\mathbf{x} \log p_t(\mathbf{x}) ||^2 ]$
• Sampling: Use Langevin dynamics or SDEs to sample from learned scores

Evaluation Metrics

Assessing generative model quality requires both quantitative and qualitative measures:

• Likelihood-based: Log-likelihood, bits-per-dimension (for flows)
• Distribution-based: Fréchet Inception Distance (FID), Kernel Inception Distance (KID)
• Sample Quality: Inception Score (IS), perceptual quality
• Diversity: Coverage, density of generated samples

Connections Between Approaches

• VAEs as Flow-like Models: Reparameterization connects to normalizing flows
• Diffusion as Hierarchical VAEs: Diffusion steps can be viewed as latent layers
• Score-Based and Diffusion: Score functions are central to both
• GANs and All: Adversarial training can be applied to any generative model

Understanding these unifying principles helps in choosing appropriate models for different applications and in developing new generative techniques.

Prerequisites: Strong background in deep learning (PyTorch/TensorFlow), probability theory, and optimization. Specifically:

Mathematical Prerequisites

• Probability Theory: Random variables, distributions (Gaussian, Bernoulli, Categorical), expectation, variance, Bayes' theorem, maximum likelihood estimation
• Information Theory: Entropy, cross-entropy, Kullback-Leibler divergence, mutual information
• Linear Algebra: Vector/matrix operations, eigenvalues/eigenvectors, singular value decomposition, tensor operations
• Calculus: Partial derivatives, chain rule, gradient descent, automatic differentiation
• Statistics: Hypothesis testing, confidence intervals, bias-variance tradeoff

Machine Learning Fundamentals

• Supervised Learning: Classification, regression, loss functions, regularization
• Neural Networks: Feedforward networks, backpropagation, activation functions, initialization
• Convolutional Networks: Convolutional layers, pooling, receptive fields, modern architectures (ResNet, Transformer)
• Optimization: Stochastic gradient descent variants (Adam, RMSProp), learning rate scheduling, batch normalization
• Regularization: Dropout, weight decay, early stopping, data augmentation

Programming and Tools

• Python: Advanced features (decorators, context managers, multiprocessing), NumPy/Pandas proficiency
• Deep Learning Frameworks: PyTorch (tensors, autograd, nn.Module, DataLoader) or TensorFlow/Keras
• Version Control: Git basics, collaborative workflows
• Development Environment: Jupyter notebooks, IDEs (VS Code, PyCharm), command-line tools

Recommended Background Reading

• "Deep Learning" by Goodfellow, Bengio, Courville (Chapters 1-5, 13-20)
• "Pattern Recognition and Machine Learning" by Bishop (Chapters 1-4, 8-10)
• "Probabilistic Machine Learning" by Murphy (Chapters 1-3, 21-24)

Students without this background may find the course challenging and are encouraged to review these topics beforehand.

Table of contents

• Course Overview
• Repository structure and purpose
• Quick start (setup & run)
• Notebooks and assignments (CA1..CA4) — summary and status
• Deep dive: CA1 (Variational Autoencoders)
• Deep dive: CA2 (GANs & Normalizing Flows)
• Deep dive: CA3 (Diffusion and Score-based Models)
• Deep dive: CA4 (Fine-Tuning Vision-Language Models)
• Data, storage and artifact management
• Reproducibility checklist and recommended configuration
• Testing and lightweight smoke checks
• Common issues and troubleshooting
• References and further reading
• Credits and license

---

Repository structure (top-level)

• CA1_Variational_Autoencoders/ — Course Assignment 1: Variational Autoencoders
  • code/ — Jupyter notebooks and code used for experiments (e.g., code.ipynb).
  • description/ — Assignment description PDF.
  • report/ — PDF reports and figures.
  • images/ — Generated images and visualizations.
  • train/ — Training datasets (CelebA subset: smile/non-smile images).
  • README.md — Detailed documentation for CA1.
• CA2_GANs_Normalizing_Flows/ — Course Assignment 2: GANs and Normalizing Flows
  • code/ — Jupyter notebooks (e.g., CA2_DGM.ipynb, Q2_final_res.ipynb).
  • description/ — Assignment description PDF.
  • report/ — PDF reports and figures.
  • images/ — Generated samples and visualizations.
  • README.md — Detailed documentation for CA2.
• CA3_Diffusion_Models/ — Course Assignment 3: Diffusion and Score-based Models
  • codes/ — Jupyter notebooks (e.g., Diffusion_Models.ipynb, score_based_models.ipynb).
  • description/ — Assignment description PDF.
  • report/ — PDF reports and figures.
  • images/ — Generated samples and visualizations.
  • README.md — Detailed documentation for CA3.
• CA4_Vision_Language_Model/ — Course Assignment 4: Vision-Language Models
  • code/ — Jupyter notebooks (e.g., final_CA4_training.ipynb, evaluation notebooks).
  • description/ — Assignment description PDF.
  • report/ — PDF reports and figures.
  • images/ — Generated images and visualizations.
  • README.md — Detailed documentation for CA4.
• Slides/ — Lecture slides and course material used in class.
  • DGM_Fall_2023_Slides/ — Course lecture slides.
  • Stanford_slides/ — Supplementary slides from Stanford's CS236 course.
  • MITSlides/ — Additional slides from MIT and other sources.
• Exams/ — Past exams and solutions.

This repository is primarily an educational resource. Notebooks are annotated for readability and (where possible) reorganized to central