Skip to content

Latest commit

 

History

History
 
 

eager

README.md

Step-by-Step

This document describes the step-by-step instructions for reproducing PyTorch BlendCNN distillation(with MRPC dataset) results with Intel® Neural Compressor.

Prerequisite

1. Environment

cd examples/pytorch/nlp/blendcnn/distillation/eager
pip install torch>=1.6.0 tqdm

2. Prepare model

Download BERT-Base, Uncased

mkdir models/ && mv uncased_L-12_H-768_A-12.zip models/
cd models/ && unzip uncased_L-12_H-768_A-12.zip

3. Prepare Datasets

Download GLUE MRPC Benchmark Datasets, After downloads dataset, you need to put dataset at ./MRPC/, list this:

ls MRPC/
dev_ids.tsv  dev.tsv  test.tsv  train.tsv

Distillation

1 Fine-tune the pretrained BERT-Base model on MRPC dataset

mkdir -p models/bert/mrpc
# fine-tune the pretrained BERT-Base model
python finetune.py config/finetune/mrpc/train.json

2 Distilling the BlendCNN with BERT-Base

Now BERT-Base model weights model_final.pt at ./models/bert/mrpc/.

mkdir -p models/blendcnn/
# distilling the BlendCNN
python distill.py --loss_weights 0.1 0.9

Follow the above steps, you will find distilled BlendCNN model weights best_model_weights.pt in ./models/blendcnn/.