which knowledge-distillation loss is best in "kd_losses "?

[yang0817manman]

thanks your shariing. Can you tell me which lossfunction is best in " "kd_losses " for classification task?

[AberHu]

Sorry for later reply. From my perspective, different kd losses are suitable for different tasks. For classification, the orginal KD (soft target) is ok, because it can be treated a variation of label smoothing regularization. You may tune the temperature and the trade-off parameters in soft target. Another tow kd losses for classification I recommended are sp and cc. Hope these kd losses have helped.