Join the community • Contribute to the library
Awesome resource collection on pruning techniques:
If you are new to pruning check the overview page on pruning or the introduction to AI optimization at this link.
Legenda:
- ✏️ 100-499 citations, ✏️✏️
$\geq$ 500 citations - ⭐ 100-249 stars, ⭐⭐
$\geq$ 250 stars
Sorting: typology / chronological / alphabetical order
- A Survey of Model Compression and Acceleration for Deep Neural Networks [✏️✏️paper]
- APQ: Joint Search for Network Architecture, Pruning and Quantization Policy [✏️CVPR][⭐⭐github]
- Compression of Deep Learning Models for Text: A Survey [✏️paper]
- Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask [✏️NIPS]
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding [✏️✏️paper]
- Dynamic Network Surgery for Efficient DNNs [✏️✏️NIPS]
- Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better Models Smaller, Faster, and Better [paper][⭐⭐github]
- ETH 2021, Sparsity in Deep Learning: Pruning + growth for efficient inference and training in neural networks: [✏️JLMR][⭐⭐github]
- Learning both Weights and Connections for Efficient Neural Networks [✏️✏️NIPS]
- Learning Efficient Convolutional Networks Through Network Slimming [✏️✏️ICCV][⭐⭐github 1.github 1.]
- Learning Filter Basis for Convolutional Neural Network Compression [✏️ICCV]
- Pruning Filters for Efficient ConvNets [✏️✏️paper]
- Pruning from Scratch [✏️AAAI]
- Recent Advances in Efficient Computation of Deep Convolutional Neural Networks [✏️paper]
- Rethinking the Value of Network Pruning [✏️✏️paper][⭐⭐github]
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks [✏️✏️paper]
- To prune, or not to prune: exploring the efficacy of pruning for model compression [✏️✏️paper]
2021
- A Probabilistic Approach to Neural Network Pruning [PMLR]
- Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework [PMLR]
- ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations [paper][⭐⭐github]
- Content-Aware GAN Compression [CVPR][⭐⭐github]
- Convolutional Neural Network Pruning with Structural Redundancy Reduction [CVPR]
- Dynamic Network Surgery for Efficient DNNs [✏️✏️NIPS][⭐⭐github]
- Group Fisher Pruning for Practical Network Compression [PMLR]
- Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation [CVPR]
- Manifold Regularized Dynamic Network Pruning [CVPR]
- Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network [paper][⭐⭐github]
- Network Pruning That Matters: A Case Study on Retraining Variants [paper][⭐⭐github]
- Network Pruning via Performance Maximization [CVPR]
- NPAS: A Compiler-aware Framework of Unified Network Pruning andArchitecture Search for Beyond Real-Time Mobile Acceleration [CVPR]
- On the Predictability of Pruning Across Scales [PMLR]
- Prune Once for All: Sparse Pre-Trained Language Models [paper][⭐⭐github]
- Pruning Neural Networks at Initialization: Why Are We Missing the Mark? [✏️paper]
- Towards Compact CNNs via Collaborative Compression [CVPR]
2020
- A Gradient Flow Framework For Analyzing Network Pruning [paper][⭐⭐github]
- A Signal Propagation Perspective for Pruning Neural Networks at Initialization [✏️paper][⭐⭐github]
- Accelerating CNN Training by Pruning Activation Gradients [ECCV]
- Adversarial Neural Pruning with Latent Vulnerability Suppression [PMLR][⭐⭐github]
- AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates [✏️AAAI]
- Bayesian Bits: Unifying Quantization and Pruning [NIPS]
- Channel Pruning via Automatic Structure Search [✏️paper][⭐⭐github]
- Comparing Rewinding and Fine-tuning in Neural Network Pruning [✏️paper][⭐⭐github]
- DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search [ECCV]
- DHP: Differentiable Meta Pruning via HyperNetworks [✏️ECCV][⭐⭐github]
- Differentiable Joint Pruning and Quantization for Hardware Efficiency [ECCV]
- Directional Pruning of Deep Neural Networks [NIPS][⭐⭐github]
- Discrete Model Compression With Resource Constraint for Deep Neural Networks [CVPR]
- DMCP: Differentiable Markov Channel Pruning for Neural Networks [✏️CVPR][⭐⭐github]
- DropNet: Reducing Neural Network Complexity via Iterative Pruning [PMLR][⭐⭐github]
- DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation [ECCV]
- Dynamic Model Pruning with Feedback [✏️paper]
- EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning [✏️ECCV][⭐⭐github]
- Fast Convex Pruning of Deep Neural Networks [paper]
- Few Sample Knowledge Distillation for Efficient Network Compression [✏️CVPR]
- Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection [✏️PMLR][⭐⭐github]
- Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression [✏️CVPR][⭐⭐github]
- HRank: Filter Pruning using High-Rank Feature Map [✏️CVPR][⭐⭐github]
- HYDRA: Pruning Adversarially Robust Neural Networks [✏️NIPS]
- Layer-adaptive Sparsity for the Magnitude-based Pruning [paper][⭐⭐github]
- Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration [✏️CVPR]
- Logarithmic Pruning is All You Need [NIPS]
- Lookahead: A Far-sighted Alternative of Magnitude-based Pruning [paper][⭐⭐github]
- Meta-Learning with Network Pruning [ECCV]
- Movement Pruning: Adaptive Sparsity by Fine-Tuning [✏️NIPS]
- Multi-Dimensional Pruning: A Unified Framework for Model Compression [CVPR]
- Neural Network Pruning with Residual-Connections and Limited-Data [✏️CVPR]
- Neural Pruning via Growing Regularization [paper][⭐⭐github]
- Neuron Merging: Compensating for Pruned Neurons [NIPS][⭐⭐github]
- Neuron-level Structured Pruning using Polarization Regularizer [NIPS]
- Operation-Aware Soft Channel Pruning using Differentiable Masks [✏️PMLR]
- Position-based Scaled Gradient for Model Quantization and Pruning [NIPS]
- ProxSGD: Training Structured Neural Networks under Regularization and Constraints [ICLR]
- Pruning Filter in Filter [NIPS]
- Pruning neural networks without any data by iteratively conserving synaptic flow [✏️NIPS]
- Reborn filters: Pruning convolutional neural networks with limited data [AAAI]
- Robust Pruning at Initialization [paper]
- Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot [NIPS]
- SCOP: Scientific Control for Reliable Neural Network PruningNeurIPSFPyTorch [✏️NIPS]
- Soft Threshold Weight Reparameterization for Learnable Sparsity [✏️PMLR][⭐⭐github]
- Storage Efficient and Dynamic Flexible Runtime Channel Pruning via Deep Reinforcement Learning [NIPS]
- Structured Compression by Weight Encryption for Unstructured Pruning and Quantization [CVPR]
- The Generalization-Stability Tradeoff In Neural Network Pruning [NIPS]
- Towards Efficient Model Compression via Learned Global Ranking [✏️CVPR][⭐⭐github]
2019
- Accelerate CNN via Recursive Bayesian Pruning [CVPR]
- Adversarial Robustness vs Model Compression, or Both? [✏️ICCV]
- Approximated Oracle Filter Pruning for Destructive CNN Width Optimization github [✏️PMLR][⭐⭐github]
- AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters [✏️NIPS]
- Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure [✏️CVPR]
- Collaborative Channel Pruning for Deep Networks [✏️PMLR]
- COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning [paper][⭐⭐github]
- DARB: A Density-Aware Regular-Block Pruning for Deep Neural Networks [paper]
- Data-Independent Neural Pruning via Coresets [✏️paper]
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis4 [✏️ICML][⭐⭐github]
- Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration [✏️✏️CVPR]
- Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks [✏️NIPS]
- Global Sparse Momentum SGD for Pruning Very Deep Neural Networks [✏️NIPS]
- Importance Estimation for Neural Network Pruning [✏️CVPR]
- MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning [✏️ICCV]
- Model Compression with Adversarial Robustness: A Unified Optimization Framework [✏️NIPS]
- Network Pruning via Transformable Architecture Search [✏️NIPS][⭐⭐github]
- OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks [✏️CVPR]
- On Implicit Filter Level Sparsity in Convolutional Neural Networks [CVPR]
- One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers [✏️NIPS]
- One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation [paper]
- Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search [✏️CVPR]
- Provable Filter Pruning for Efficient Neural Networks [✏️paper][⭐⭐github]
- Structured Pruning of Neural Networks with Budget-Aware Regularization [✏️CVPR]
- The State of Sparsity in Deep Neural Networks [✏️paper][⭐⭐github]
- Towards Optimal Structured CNN Pruning via Generative Adversarial Learning [✏️CVPR]
- Variational Convolutional Neural Network Pruning [✏️CVPR]
2018
- A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers [✏️ECCV][⭐⭐github]
- Accelerating Convolutional Networks via Global & Dynamic Filter Pruning [✏️NIPS]
- Amc: Automl for model compression and acceleration on mobile devices [✏️✏️ECCV][⭐⭐github]
- CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-Quantization [✏️CVPR]
- Constraint-Aware Deep Neural Network Compression [✏️ECCV]
- Coreset-Based Neural Network Compression [✏️ECCV]
- Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds [✏️paper]
- Data-Driven Sparse Structure Selection for Deep Neural Networks [✏️ECCV][⭐⭐github]
- Discrimination-aware Channel Pruning for Deep Neural Networks [✏️NIPS]
- Dynamic Channel Pruning: Feature Boosting and Suppression [✏️paper][⭐⭐github]
- Dynamic Sparse Graph for Efficient Deep Learning [paper]
- Frequency-Domain Dynamic Pruning for Convolutional Neural Networks [✏️NIPS]
- “Learning-Compression” Algorithms for Neural Net Pruning [✏️CVPR]
- Learning Sparse Neural Networks via Sensitivity-Driven Regularization [✏️NIPS]
- Model Compression and Acceleration for Deep Neural Networks [✏️IEEE]"
- NISP: Pruning Networks using Neuron Importance Score Propagation [✏️✏️CVPR]
- PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning [✏️✏️CVPR][⭐⭐github]
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers [✏️paper][⭐⭐github]
- SNIP: Single-shot Network Pruning based on Connection Sensitivity [✏️✏️paper][⭐⭐github]
- Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks [✏️✏️paper][⭐⭐github]
- Stronger generalization bounds for deep nets via a compression approach [✏️PMLR]
2017
- Channel pruning for accelerating very deep neural networks [✏️✏️ICCV][⭐⭐github]
- Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning [✏️✏️CVPR]
- Efficient Processing of Deep Neural Networks: A Tutorial and Survey [✏️✏️IEEE]
- Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon [✏️NIPS]
- Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee [✏️NIPS]
- Runtime Neural Pruning [✏️NIPS]
2016
- Pruning Convolutional Neural Networks for Resource Efficient Inference [✏️✏️paper]
1989
- Optimal Brain Damage (LeCun) [✏️✏️NIPS]
Courses
- Stanford 2020, Hardware Accelerators for Machine Learning [paper]
Webinars, video content
- Cornell 2022, ML HW & Systems [paper]
- MIT 2021, Vivienne Sze & Lex Fridman, Efficient Computing for Deep Learning, Robotics, and AI [paper]
- Stanford 2017, Son Han, Efficient Methods and Hardware for Deep Learning [paper]
Blogs, written content
- Adding Quantization-aware Training and Pruning to the TensorFlow Model Garden [paper]
- A friendly introduction to machine learning compilers and optimizers [paper]
- Faster Deep Learning Training with PyTorch – a 2021 Guide [paper]
- Optimize machine learning models with Tensorflow [paper]