Knowldege distillation

[Leila-Malihi]

Hi,
Could you please tell me this algorithm you implemented for knowledge distillation is which one? I mean there is some like: Fitnet,AT, FT,..

[yoshitomo-matsubara]

Hi @Leila-Malihi ,

If you're asking about what method is specifically referred to as "Knowledge Distillation", it's Hinton et al. (2014)'s knowledge distillation (KD).
FitNet, AT, FT, and more also fall into the category of knowledge distillation

Close this discussion and "Mark as answer" if it answers your question