why is MSE of prediction way different from loss over batches

Question

I am new to machine learning so forgive me if i ask stupid question. I have a time series data and i split it into training and test set.

This is my code:

from numpy import array
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
    X, y = list(), list()
    for i in range(len(sequence)):
    # find the end of this pattern
        end_ix = i + n_steps_in
        out_end_ix = end_ix + n_steps_out
        # check if we are beyond the sequence
        if out_end_ix > len(sequence):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

# choose a number of time steps
n_steps_in, n_steps_out = 10, 5
# split into samples
X, y = split_sequence(trainlist, n_steps_in, n_steps_out)
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mean_squared_error')
# fit model
history = model.fit(X, y, epochs=2000, verbose=0)
# demonstrate prediction
x_input = array(testlist[0:10])
x_input = x_input.reshape((1, n_steps_in))
yhat = model.predict(x_input, verbose=0)
yhat=list(yhat[0])

when i do print(history.history['loss'][-10:-1]) it gives me roughly 0.55 and when i do

from sklearn.metrics import mean_squared_error 
mean_squared_error(testlist[11:16],yhat)

it gives me 0.11. Why is it so different?

score 0 · Accepted Answer · answered Nov 28 '19 at 20:04

From my understanding, you are comparing the prediction error on the test set versus the training loss error. When the training error is greater than the test error, that means your model is under-fitted.

Under-fitting occurs when there is still room for improvement on the test data. This can happen for a number of reasons: If the model is not powerful enough, is over-regularized, or has simply not been trained long enough. This means the network has not learned the relevant patterns in the training data. There are number of ways you can resolve under-fitting problem:

Increase the time of training procedure
Increase number of parameters in your model or the complexity of your model

why is MSE of prediction way different from loss over batches

1 Answers1