Scikit learn linear regression - learning rate and epoch adjustment

Question

I am trying to learn linear regression using ordinary least squares and gradient descent from scratch.

I read the documentation for the Scikit learn function and I do not see a means to adjust the learning rate or the epoch with the sklearn.linear_model.LinearRegression class.

Is there a standard learning rate for the linear regression model?

Epochs I am assuming are determined by the change in the error function and are thus dependent on the dataset, with some predetermined maximum amount before convergence.

Are there any other good packages that have a linear regression model that I can test my own model against with adjustable learning rates and epochs?

score 1 · Accepted Answer · answered Sep 05 '20 at 08:50

A linear regression model $y=\beta X+u$ can be solved in one "round" by using $(X'X)^{-1}X'y=\hat{\beta}$. It can also be solved using gradient descent but there is no need to adjust something like a learning rate or the number of epochs since the solver (usually) converges without much trouble.

Here is a minimal example in R:

x0 <- c(1,1,1,1,1) 
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))

x
y

# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y 

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))
  return(t(gradient))
}

# define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters
  
  alpha = .05 # set learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)   
  }
  return(theta)
}

# results without feature scaling
print(grad.descent(x,2000))
beta1
beta2

Thank you, I wrongly assumed that the scikitlearn system uses gradient descent by default. Ordinary least squares only works with a matrix that you can invert right? Otherwise it won't be solvable? — chrisper, Sep 06 '20 at 09:17
I‘m not sure what solver is used with scikit. However OLS can be solved by the standard formula, which requires that X can be inverted. To see the difference between OLS and regression with regulation (as in the SGD case), see ISL, Chapter 6.2., i.e. equation 6.5. Regulation is introduced by an additional penalty term on top of the standard OLS loss function. So the results OLS vs. Lasso etc. are different http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf#page224 — Peter, Sep 06 '20 at 09:43

score 0 · Answer 2 · answered Sep 05 '20 at 05:32

0

You can use SGDRegressor available in scikit learn for adjusting learning rate. It has a variety of parameters you can adjust.

answered Sep 05 '20 at 05:32

Ankit Seth

1,801
12
27

This is a lasso/ridge/elastic net regressor, not OLS as specified in the question. – Peter Sep 05 '20 at 10:36
@Peter OP's first line says OLS and Gradient Descent and he never said anything about a specific algorithm. OP just wants a Linear Regression model with adjustable learning rate, so I suggested SGD regression. – Ankit Seth Sep 06 '20 at 05:47
OLS is not lasso/ridge/elastic net since the loss function is different. – Peter Sep 06 '20 at 08:37
@Ankit Seth thank you. I will try out the sgd regressor and see how the results compare. – chrisper Sep 06 '20 at 09:19

Scikit learn linear regression - learning rate and epoch adjustment

2 Answers2