How to add a sparse loss during training

[rogercmq]

I am working on adding a L1-norm loss term during training, aiming to obtain sparse network weights. Could you give me some advice?

[congee524]

Since we calculated loss in the HEAD, I think the best way is to modify the optimizer.....

For example, torch.optim.SGD directly computes the gradient of L2 regularization (1/2 w^2 ---> w):

if weight_decay != 0:
    d_p = d_p.add(p, alpha=weight_decay)

Maybe you can

d_p = d_p.add(sign(p), alpha=weight_decay_l1)

[dreamerlin]

It has been answered @rogercmq

[rogercmq]

Problem solved. Thx!