Efficient Diffusion Models: A Survey

If you like our project, please give us a star ⭐ on GitHub for the latest update.

Diffusion models have kickstart a new era in the field of artificial intelligence generative content (AIGC). This repo is a curated list of papers about the latest advancements in efficient diffusion models. This repo is being actively updated, please stay tuned!

📣 Update News

[2024-11-9] We released the repository.

❤️ Community Support

We will actively maintain this repository by incorporating new research as it emerges. If you have any suggestions regarding our taxonomy, find any missed papers, or update any preprint arXiv paper that has been accepted to some venue, feel free to send us an email or submit a pull request using the following markdown format.

📖 Table of Contents

• Efficient Diffusion Models: A Survey
  • 📣 Update News
  • ❤️ Community Support
  • 📖 Table of Contents
    • Algorithm
      • Efficient Training
      • Efficient Fine-tuning
      • Efficient Sampling
      • Compression
    • System
      • Optimized Hardware-Software Co-Design
      • Parallel Computing
      • Caching Technique
    • Frameworks
  • 👍 Acknowledgement
  • 📑 Citation

Algorithm

Efficient Training

• Noise Schedule

  • [NeurIPS 2020] Denoising Diffusion Probabilistic Models. [Paper] [Code]
  • [ICLR 2021] Denoising Diffusion Implicit Models. [Paper]
  • [ICML 2021] Improved Denoising Diffusion Probabilistic Models. [Paper] [Code]
  • [Arxiv 2024.07] Improved Noise Schedule for Diffusion Training. [Paper]
  • [EMNLP 2023] A Cheaper and Better Diffusion Language Model with Soft-Masked Noise. [Paper] [Code]
  • [NeurIPS 2024] ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting. [Paper] [Code]
  • [Arxiv 2024.06] Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment. [Paper] [Code]
  • [ICLR 2023] DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models. [Paper] [Code]
  • [ACL 2024] Text Diffusion Model with Encoder-Decoder Transformers for Sequence-to-Sequence Generation. [Paper]

• Score Matching

  • [JMLR 2005] Estimation of Non-Normalized Statistical Models by Score Matching. [Paper]
  • [UAI 2019] Sliced Score Matching: A Scalable Approach to Density and Score Estimation. [Paper] [Code]
  • [NeurIPS 2019] Generative Modeling by Estimating Gradients of the Data Distribution. [Paper] [Code]
  • [NeurIPS 2021] Maximum Likelihood Training of Score-Based Diffusion Models. [Paper] [Code]
  • [ICLR 2022] Score-Based Generative Modeling with Critically-Damped Langevin Diffusion. [Paper] [Code]

• Data-Dependent Adaptive Priors

  • [Arxiv 2022.02] PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior. [Paper]
  • [Arxiv 2023.05] DiGress: Discrete Denoising diffusion for graph generation. [Paper] [Code]
  • [Arxiv 2024.02] DecompDiff: Diffusion Models with Decomposed Priors for Structure-Based Drug Design. [Paper] [Code]
  • [CVPR 2023] Leapfrog diffusion model for stochastic trajectory prediction. [Paper] [Code]

• Rectified Flow

  • [Arxiv 2022.09] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. [Paper] [Code]
  • [ICLR 2024] InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation. [Paper] [Code]
  • [Arxiv 2022.09] Rectified Flow: A Marginal Preserving Approach to Optimal Transport. [Paper]
  • [Arxiv 2024.10] Improving the Training of Rectified Flows. [Paper] [Code]
  • [Arxiv 2023.09] Diffusion Models with Deterministic Normalizing Flow Priors. [Paper] [Code]
  • [Arxiv 2024.09] PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Accelerator. [Paper] [Code]
  • [Arxiv 2024.02] SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flow. [Paper] [Code]

Efficient Fine-tuning

• Low Rank Adaptation

Efficient Sampling

• Solver

  • [ICLR 2021] Score-Based Generative Modeling through Stochastic Differential Equations. [Paper] [Code]
  • [NeurIPS 2021] Diffusion Normalizing Flow. [Paper]
  • [Arxiv 2021.05] Gotta Go Fast When Generating Data with Score-Based Models. [Paper] [Code]
  • [NeurIPS 2023] Gaussian Mixture Solvers for Diffusion Models. [Paper] [Code]
  • [ICML 2024] Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equations. [Paper] [Code]
  • [NeurIPS 2023] SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models. [Paper]
  • [NeurIPS 2022] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. [Paper] [Code]
  • [ICLR 2023] Fast Sampling of Diffusion Models with Exponential Integrator. [Paper] [Code]
  • [ICML 2023] Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs. [Paper] [Code]
  • [ICML 2023] Denoising MCMC for Accelerating Diffusion-Based Generative Models. [Paper] [Code]

• Efficient Scheduling

  • [NeurIPS 2022] Deep Equilibrium Approaches to Diffusion Models. [Paper] [Code]
  • [ICML 2024] Accelerating Parallel Sampling of Diffusion Models. [Paper] [Code]
  • [NeurIPS 2023] Parallel Sampling of Diffusion Models. [Paper] [Code]

• Partial Sampling

  • [EMNLP 2021] Consistent Accelerated Inference via Confident Adaptive Transformers. [Paper] [Code]
  • [NeurIPS 2022] Confident Adaptive Language Modeling. [Paper]
  • [ICML 2024] A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models. [Paper] [Code]
  • [Arxiv 2022.04] Semi-Parametric Neural Image Synthesis. [Paper] [Code]
  • [ICLR 2023] kNN-Diffusion: Image Generation via Large-Scale Retrieval. [Paper]
  • [ICLR 2023] Re-Imagen: Retrieval-Augmented Text-to-Image Generator. [Paper]
  • [ICML 2023] ReDi: Efficient Learning-Free Diffusion Inference via Trajectory Retrieval. [Paper] [Code]

Compression

• Quantization

  • [CVPR 2023] Post-training Quantization on Diffusion Models. [Paper] [Code]
  • [ICCV 2023] Q-Diffusion: Quantizing Diffusion Models. [Paper] [Code]
  • [ICLR 2021] BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction. [Paper] [Code]
  • [NeurIPS 2023] Leveraging Early-Stage Robustness in Diffusion Models for Efficient and High-Quality Image Synthesis. [Paper]
  • [NeurIPS 2023] PTQD: Accurate Post-Training Quantization for Diffusion Models. [Paper] [Code]
  • [NeurIPS 2023] Temporal Dynamic Quantization for Diffusion Models. [Paper]
  • [ICLR 2020] Learned Step Size Quantization. [Paper]
  • [ICLR 2024] EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models. [Paper] [Code]

• Pruning

  • [NeurIPS 2023] Structural Pruning for Diffusion Models. [Paper] [Code]
  • [CVPRW 2024] LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights. [Paper]
  • [ICML 2024] LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging. [Paper] [Code]
  • [Arxiv 2024.04] LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models. [Paper]

• Knowledge Distillation

  • [Arxiv 2021.01] Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed. [Paper] [Code]
  • [ICLR 2022] Progressive Distillation for Fast Sampling of Diffusion Models. [Paper] [Code]
  • [CVPR 2023] On Distillation of Guided Diffusion Models. [Paper]
  • [ICML 2023] Consistency Models. [Paper] [Code]
  • [ECCV 2020] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. [Paper] [Code]
  • [ICLR 2023] DreamFusion: Text-to-3D using 2D Diffusion. [Paper] [Project]
  • [NeurIPS 2023] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. [Paper] [Code]
  • [NeurIPS 2023] Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models. [Paper] [Code]
  • [CVPR 2024] 3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation. [Paper] [Code]

System

Optimized Hardware-Software Co-Design

• [CVPRW 2023] Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations. [Paper] [Project]
• [FPL 2024] SDA: Low-Bit Stable Diffusion Acceleration on Edge FPGAs. [Paper] [Code]
• [ISCAS 2024] A 28.6 mJ/iter Stable Diffusion Processor for Text-to-Image Generation with Patch Similarity-based Sparsity Augmentation and Text-based Mixed-Precision. [Paper]

Parallel Computing

• [CVPR 2024] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. [Paper] [Code]
• [Arxiv 2024.05] PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Diffusion Transformer Models. [Paper] [Code]
• [Arxiv 2024.07] SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modules. [Paper]
• [MLSys 2024] DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines. [Paper]

Caching Technique

• [NSDI 2024] Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models. [Paper] [Code]
• [CVPR 2024] DeepCache: Accelerating Diffusion Models for Free. [Paper] [Code]
• [CVPR 2024] Cache Me if You Can: Accelerating Diffusion Models through Block Caching. [Paper] [Project]
• [Arxiv 2024.06] Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching. [Paper] [Code]
• [Arxiv 2024.07] FORA: Fast-Forward Caching in Diffusion Transformer Acceleration. [Paper] [Code]

Frameworks

	Efficient Training	Efficient Fine-Tuning	Efficient Inference
Diffusers [Code]	✅	✅	✅
DALL-E [Code]	❌	❌	✅
OneDiff [Code]	❌	❌	✅
LiteGen [Code]	✅	✅	✅
InvokeAI [Code]	❌	✅	✅
ComfyUI-Docker [Code]	❌	✅	✅
Grate [Code]	❌	✅	✅
Versatile Diffusion [Code]	✅	✅	✅
UniDiffuser [Code]	✅	✅	✅