L1 regularization implementation

[ColdTeapot273K]

Hello! My application requires for L1 and I've been trying to implement at least the simplest case of L1 regularization for LinearRegression from #76 , but to no effect.

As far as I can tell, the main stuff happens at glm.py line 94, although I don't understand why L2 norm doesn't seem to be applied.
I assumed weights are somehow already squared there, anyway, taking sqrt from self._weights gives imaginary numbers and taking vectorized sign from the VectorDict is non-trivial.

I'm good with any quick hack to implement the L1

[gbolmier]

Hi @ColdTeapot273K! Have you tried to reproduce the L1 cumulative and the Truncated gradient techniques discussed here in #76 (cf. implementation)? If yes, I would say it's probably just a matter of hyper-parameter tuning, try playing with λ, K, and θ

[ColdTeapot273K]

Hello @gbolmier , that's what I was trying to do, but natively, inside the glm module, to leverage stuff like Nadam optimizer from optim module (and didn't succeed).

Ended up using estimators of yours as standalone classes called outside the pipeline, without fancy optimizers or losses. Does the job kinda.

[gbolmier]

Hey @ColdTeapot273K 👋 If you have the time, you should open a pull request. Then we could discuss, debug, and test the implementation against different datasets. Cheers!