Training problems for JHMDB datasets

[huang-chenhai]

I used a pre-training dataset that worked fine during training and did not predict correct results on the validation set。
To my surprise, everything works fine when continuing training with the weights provided by the author that have already been trained（TubeR_CSN152_JHMDB.pth）. But training from scratch will cause problems like the code below.

Train
Epoch: [1][460/2839] lr: 5e-05 data_time: 0.025, batch time: 0.657 class_error: 11.761, loss: 7.271, loss_bbox: 0.085, loss_giou: 0.161, loss_ce: 0.356, loss_ce_b: 0.000
Epoch: [1][461/2839] lr: 5e-05 data_time: 0.031, batch time: 0.661 class_error: 11.736, loss: 7.273, loss_bbox: 0.085, loss_giou: 0.161, loss_ce: 0.356, loss_ce_b: 0.000

eval
Epoch: [0][9138/9139] data_time: 0.003, batch time: 0.069 class_error: 99.354, loss: 15.582, loss_bbox: 0.143, loss_giou: 0.233, loss_ce: 1.298
Epoch: [0][9139/9139] data_time: 0.003, batch time: 0.068 class_error: 99.354, loss: 15.582, loss_bbox: 0.143, loss_giou: 0.233, loss_ce: 1.298

per_class_len 24 per_class [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.02090471 0. nan nan nan]

[Yanruyu-Beta]

I have the same problem