Relation between "underfitting" vs "high bias and low variance"

Question

What is the exact relation between "underfitting" vs "high bias and low variance". They seem to be tightly related concepts but still 2 distinct things. What is the exact relation between them?

Same for "overfitting" vs "high variance and low bias". The same, but not exactly?

e.g.

Wikipedia states that:

"The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting). The variance is an error from sensitivity to small fluctuations in the training set. High variance may result from an algorithm modeling the random noise in the training data (overfitting)."

Kaggle says:

"In a nutshell, Underfitting – High bias and low variance"

towardsdatascience:

"In simple terms, High Bias implies underfitting"

or another question from Stack exchange:

"underfitting is associated with high bias and over fitting is associated with high variance"

Answer 1

Underfitting and high bias are closely related concepts in machine learning. Underfitting refers to a model that is not complex enough to capture the underlying trend in the data. It is not able to capture the patterns in the data well and as a result, it performs poorly on both the training and the test sets.

On the other hand, high bias refers to the tendency of a model to consistently make the same types of errors, regardless of the input data. A model with high bias pays little attention to the training data and oversimplifies the model, leading to poor performance on the training and test sets.

So, underfitting is often caused by a model with high bias, which means that it is oversimplifying the problem and is not able to capture the complexity of the data.

Overfitting, on the other hand, refers to a model that is too complex and fits the noise in the training data, rather than the underlying trend. It performs well on the training set, but poorly on the test set.

High variance is often a cause of overfitting, as it refers to the sensitivity of the model to small fluctuations in the training data. A model with high variance pays too much attention to the training data and ends up learning the noise in the data, rather than the underlying trend.

Therefore, overfitting is often caused by a model with high variance, which means that it is too sensitive to the noise in the training data and is not able to generalize well to unseen data.

In short, underfitting is usually caused by high bias, which leads to oversimplification of the model and poor performance on both the training and the test sets. On the other hand, overfitting is usually caused by high variance, which leads to a model that is too sensitive to the noise in the training data and performs poorly on the test set.

Answer 2

I find this graph helpful:

The X-axis here is a model complexity. The complexity is measured by the amount of weights/decisions/parameters that the model includes. In decision trees, it can be interpreted as tree depth. In neural networks, it is comparable to the number of weights (additional cells/hidden layers/other connections that require fitting).

Let's stick with the DT example and assume that we have two models: a shallow tree (e.g. depth of 2) and a deep one (e.g. depth of 50).

Because the shallow tree has a small number of decision rules that it can hold, no matter how much data we train it on its result may be far away from truth. This model has high bias error because it is simply not complex enough.

If we train our deep tree with just a couple of records in a train set, we may find ourselves in the same area of a high bias, which is caused by underfitting - the model contains enough parameters to train but lacks data.

Overfitting happens when the model is too complex for the amount of the train data. Parameters a fitted too much, which is expressed in significant (variance) errors of unseen datasets. We could rather train less (e.g. reduce the number of epochs), reduce the amount of train data (e.g. undersample for imbalanced problems), reduce the number of parameters (e.g. prune the tree, add dropout) or add penalty constraints (e.g. L1, L2).