Training Time

[meltingCat]

HI, I would like to know how many time did you train on an epoch.
I spent an hour to train an epoch using RTX2080. Doesn't seem right to deal with such small dataset.

---------------Epoch 1-------------
---------Saving checkpoint---------
Testing: 100%|████████████████████████████████| 124/124 [03:28<00:00, 1.68s/it]
Error Rate L0 0.315 Error Rate L1 0.0407 Error Rate L2 0.0249 Error Rate L3 0.0246
Epoch 1: 100%|████████████████████████████████| 753/753 [54:31<00:00, 4.34s/it]
Loss [ 28.6935 6.8569 4.2865 895.1974]
Activity Rate [95.05911542127708, 49.73174018810146, 38.44802363960695, 66.41492680630824]
Changing learning rate from 1e-09 to 1e-09

[eneftci]

Which dataset is this? and what are the dimensions of your input [batch_size, timesteps, channels, height, width]? 4s per iteration seems reasonable.

[meltingCat]

I ran “train_lenet_decolle.py” with default setting and Mnist . Maybe I should increase the batch size, then try again.

[eneftci]

Unfortunately this is normal. N-MNIST is a bit long to train.