Soft Labels in SRe2L

[qwrazdf]

Dear Authors,

Thank you for your excellent work. I have a question regarding the use of additional soft labels in SRe2L. I am concerned whether this might lead to unfairness and whether it conflicts with the objectives of DD.

I look forward to your response. Once again, thank you for your outstanding contribution.

[szq0214]

Hi @qwrazdf Thanks for your interest in our work.

We emphasize the distillation of dataset (DD) includes both the images and the corresponding soft or hard labels as integral components. Our soft labels are independent of the teacher model during post-evaluation or actual usage, ensuring that no information from the original dataset is involved in this stage. This is why we use FKD instead of conventional KD to generate soft labels. As a result, our setup is entirely fair and reasonable.

[qwrazdf]

Thank you for your prompt response. Could it be understood that this approach trades off some storage cost to ensure downstream training efficiency and outstanding performance?

[szq0214]

Hi @qwrazdf Yes, you can think of it that way, but the storage cost for the soft labels can be significantly minimized, there are several label compression/quantization strategies discussed in FKD paper. Of course, in DD scenario, this could cause some performance degradation.