Remove `intercept_` attribute? about lightning HOT 17 CLOSED

Is it just to maintain compatibility over sklearn's predict method?

from lightning.

Not a big deal. This was just to maintain attribute compatibility with scikit-learn.

from lightning.

@mblondel I just want to quickly clarify. The algorithm implemented in lightning is Algorithm 1 in page 4 of this paper http://jmlr.org/papers/volume13/yuan12a/yuan12a.pdf right?

from lightning.

Yes but lightning supports coordinate descent without line search too.

from lightning.

Is it done by using max_steps=0, because I remember you mentioning, "It adds a lot of code, with no speed benefit"

from lightning.

More specifically this comment, scikit-learn/scikit-learn#2788 (comment) . It seems to me from the documentation that, max_step=0 uses a constant step size. However in the code, it seems that the lipschitz constants are found when the max_step=0. So I was getting a bit confused.

from lightning.

Yes, line search is complicated to implement and there's no big benefit in my experience. Have look at Fig. 2 in my paper for a comparison in the case of group lasso. Line search makes a huge difference for full gradient descent algorithms though.

from lightning.

Yes CD without line search is enabled when max_steps=0. The inverse of the Lipschitz constants are the step sizes for each feature / block. Are you working on a CD solver for scikit-learn?

from lightning.

Ok, I got confused.

Are you working on a CD solver for scikit-learn?

Yes. It was initially part of my GSoC but then it became slightly too much to do. So I am working on it now.

from lightning.

It would be more interesting for me if you could work on something which is not already in lightning (and in scikit-learn).

from lightning.

Actually, the main purpose is Elastic Net regularization for Logistic Regression. I find that only the group lasso penalty has been implemented in lightning. I am using the l1 regularization code in lightning as a base. Does that count?

from lightning.

Indeed, elastic-net is missing from lightning. There is however debiasing which consists in retraining using l2 regularization after the lasso.

from lightning.

So I suppose it counts as different?

from lightning.

You don't need my permission to work on whatever you like :)

from lightning.

@mblondel I'm sorry to keep bugging you, but I promise you it will be only until, I understand the descent code fully.

For your computation of derivatives in https://github.com/mblondel/lightning/blob/master/lightning/impl/primal_cd_fast.pyx#L1019 does b[i] somehow play the role of y[i] * (w'*X[i]). If yes, could you tell me where, because I cannot seen it be overwritten anywhere except when set to np.ones(1, n_features) initially. Or is it something else?

from lightning.

It's updated here https://github.com/mblondel/lightning/blob/master/lightning/impl/primal_cd_fast.pyx#L1045

from lightning.

I see thanks.

from lightning.