I was applying different regression models to Kaggle Housing dataset for advanced regression. I am planning to test out lasso, ridge and elastic net. However, none of these models have learning rate as their parameter.
How is the learning rate calculated for these model? Is it dependent on the dataset being trained? I know these models are regularized linear regression and must use learning rate to update their model weights. Or is their a different way to update the model?
With sklearn you can have two approaches for linear regression:
1) LinearRegression object uses Ordinary Least Squares (OLS) solver from scipy, as Learning rate (LR) is one of two classifiers which have closed form solution. This is achieve by just inverting and multiplicating some matrices.
2) SGDRegressor which is an implementation of stochastic gradient descent, very generic one where you can choose your penalty terms. To obtain linear regression you choose loss to be L2 and penalty also to none (linear regression) or L2 (Ridge regression)
There is no "typical gradient descent" because it is rarely used in practise. If you can decompose your loss function into additive terms, then stochastic approach is known to behave better (thus SGD) and if you can spare enough memory - OLS method is faster and easier (thus first solution).
This answer is mostly extracted from https://stackoverflow.com/questions/34469237/linear-regression-and-gradient-descent-in-scikit-learn-pandas/34470001#34470001 The OP @Netro is and the answer is from @lejlot
