Table of Contents

• ε-ResNet

• Installation

• Datasets

• Training

  • On ImageNet
  • On other datasets

• Testing

  • Compressing model
  • Testing compressed model

• Implementation

• Citing ε-ResNet

ε-ResNet

ε-ResNet is a variant of ResNet to automatically discard redundant layers, which produces responses that are smaller than a threshold ε, with a marginal or no loss in performance.

Installation

Dependency:

• Python 2.7

# Dependency
pip install -r requirements.txt

# Pull tensorpack
git clone https://github.com/ppwwyyxx/tensorpack.git

# This implementation is based on tags/0.2.0. I noticed that some APIs in the latest tensorpack is changed, I'll make it compatible in the future version.  
cd tensorpack && git checkout tags/0.2.0 && cd ..

# Pull epsilon-ResNet
git clone https://github.com/yuxwind/epsilonResnet.git

# Copy the adaptive learning rate file to tensorpack 
cp epsilonResnet/scripts/LearningRateSetter.py tensorpack/tensorpack/callbacks

# Run scripts
cd epsilonResnet/scripts

Datasets

ε-ResNet is tested on four datasets: CIFAR-10, CIFAR100, SVHN, ImageNet.

If you would like to compare with our experiments in your research, please run the following scripts directly.

• run_cifar10.sh
• run_cifar100.sh
• run_imagenet.sh
• run_svhn.sh

A few trained and compressed models on CIFAR-10 and ImageNet are provided here

Training

On ImageNet

Here is an example of training on ImageNet with two variants of ResNet 101: Pre-activation ResNet(the standard one) and ε-ResNet.

Two ε values 2.0 and 2.1 give out 20.12% and 25.60% compression ratio separately.

Usage:

python imagenetEpsilonResnet.py -d 101 -e 2.0 --gpu 0,1,2,3 --data {path_to_ilsvrc12_data}  

On other datasets

Here is an example of training on CIFAR-10 with ResNet-110:

• Pre-activation ResNet(the orange line)
• Pre-activation ResNet + side supervision(the purple line)
• ε-ResNet(the blue line， ε=2.5)

Their learning rates are as below. The two baselines adopt the same learning rate policy.

Usage:

python cifarEpsilonResnet.py -n 18 -e 2.5 --gpu 1 -o cifar10-e_2.5-n_18 

Note: The usage of training CIFAR100, SVHN are similar. Please refer to their scripts for examples.

Testing

Compressing model

We do testing on an standard ResNet after discarding the redundant layers.

Usage:

python compressModel.py --dir models/cifar10-e_2.5-n_125 --step 303420

Parameters:

 --dir:  specifies the train_log file directory. Three files are required. They are for this example
 	log.log
 	model-303420.data-00000-of-00001
 	model-303420.index
 --step:  specifies the model of which step is to be compressed.

It will generate compressed model files:

compressed_model_303420.cfg.
compressed_model_303420.data-00000-of-00001
compressed_model_303420.index

Testing compressed model

Here is an example to test on CIFAR-10 dataset. The script cifarCompressedResnet.py builds a standard ResNet. It reads '*.cfg' file to get the structure of the compressed model.

Usage:

python cifarCompressedResnet.py --cfg models/cifar10-n_125/compressed_model_303420.cfg --gpu 0 --cifar10 

Implementation

Please refer to here for more details of implementation.

Citing ε-ResNet

Please cite ε-ResNet in your publication if it helps your research:

@InProceedings{Yu_2018_CVPR,
author = {Yu, Xin and Yu, Zhiding and Ramalingam, Srikumar},
title = {Learning Strict Identity Mappings in Deep Residual Networks},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}