1

Does the appliance of R-squared to non-linear models depends on how we calculate it? $R^2 = \frac{SS_{exp}}{SS_{tot}}$ is going to be an inadequate measure for non-linear models since an increase of $SS_{exp}$ doesn't necessarily mean that the variance is decreasing, but if we calculate it as $R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$, then it's as much meaningful for non-linear models as it is for linear ones. I asked a similar question here where I showed that R-squared is no worse for non-linear models

So, what is the particular reason to say that in the case of non-linear models $\mathbf{R^2}$ loses its interpretation of proportion of variance explained? In both cases (I mean linear and non-linear models) we learn by how much your model's variance decreased with respect to its initial (total) variance. If $R^2 = 0.86 \text %$ then your model's variance decreased by $0.86 \text %$ (no matter whether it's linear or not).

EDIT:

  1. $SS_{tot} = \|y - \bar y\|^2$
  2. $SS_{exp} = \|\hat y - \bar y\|^2$
  3. $SS_{res} = \|y - \hat y\|^2$.

Where:

  1. $y$ is a vector of true answers;
  2. $\bar y$ is a vector whose elements are mean of $y$;
  3. $\hat y$ is a vector with our model's predictions.
mathgeek
  • 121
  • 3
  • What is $SS_{exp}?$ – Dave Oct 11 '21 at 23:48
  • @Dave, Hello! I've edited. – mathgeek Oct 12 '21 at 00:33
  • In a linear model, is $SS_{tot}=SS_{res}+SS_{exp}?$ What about in a nonlinear model? https://stats.stackexchange.com/q/427390/247274 https://stats.stackexchange.com/questions/494274/why-does-regularization-wreck-orthogonality-of-predictions-and-residuals-in-line – Dave Oct 12 '21 at 00:38
  • @Dave, Yes, I see that. But do you actually calculate $SS_{exp}$ when you use $R^2$? No. The formula you use is $R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$ which doesn't care about $SS_{exp}$ at all. It just calculate the difference $SS_{tot} - SS_{res}$ in the nominator and after that divides it by $SS_{tot}$. Why is it worse for nonlinear models, then? We still learn by how much your model's variance decreased with respect to its initial (total) variance. – mathgeek Oct 12 '21 at 00:47
  • Doesn’t the “exp” mean “explained”? – Dave Oct 12 '21 at 00:48
  • @Dave, Yes, it does. – mathgeek Oct 12 '21 at 00:49
  • It has that name because it is the sum of squares explained by the regression. When you lack the orthogonality discussed on my links, the explained sum of squares is not equal to $SS_{tot}-SS_{res}$, which is the case when we have such orthogonality. – Dave Oct 12 '21 at 01:06
  • @Yes, I know that, but what is that orthogonality for? I mean, yes, with orthogonality you have $SS_{exp} = SS_{tot} - SS_{res}$. But ***What does R-squared calculate?*** I mean, what is it? It's just the portion of reduced variance **over** the total amount of variance. But it is **always the same result** when you calculate R-squared for non-linear models by the formula $R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$. – mathgeek Oct 12 '21 at 01:11
  • @Dave, When you do it for non-linear models you still find **the portion of reduced variance** over **the total amount of variance**, just because **the portion of reduced variance** is equal to $SS_{tot} - SS_{res}$ no matter whether your model is linear or not. – mathgeek Oct 12 '21 at 01:13

0 Answers0