H2O deep learning model performance

Question

I am discovering H2O deeplearning and I would like to have your point of view about the performance that's performed my model on classification problem. Do you think my model is overfitting?

dl_fit2 <- h2o.deeplearning(x = predictors, y = response,
                           training_frame = train,
                           validation_frame = valid,
                           epochs = 200,
                           score_validation_samples=10000,
                           score_duty_cycle=0.025,
                           activation = "RectifierWithDropout",
                           hidden = c(80, 10, 80),
                           hidden_dropout_ratios = c(0.2, 0.2, 0.2),
                           loss = "CrossEntropy",
                           rate=0.01,
                           rate_annealing=2e-6,
                           adaptive_rate = FALSE,
                           momentum_start = 0.2,
                           momentum_ramp = 1e7,
                           momentum_stable = 0.4,
                           nesterov_accelerated_gradient = TRUE,
                           l1 = 1e-5,
                           l2 = 1e-5,
                           max_w2=10
)


MSE:  0.009757329
RMSE:  0.09877919
LogLoss:  0.03527449
Mean Per-Class Error:  0.01219048
AUC:  0.9991871
pr_auc:  0.4974259
Gini:  0.9983743

Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
          NO   YES    Error        Rate
NO     10022   140 0.013777  =140/10162
YES      109 10170 0.010604  =109/10279
Totals 10131 10310 0.012181  =249/20441

Maximum Metrics: Maximum metrics at their respective thresholds
                        metric threshold    value idx
1                       max f1  0.367137 0.987906 212
2                       max f2  0.178219 0.990603 270
3                 max f0point5  0.765904 0.990982 122
4                 max accuracy  0.371864 0.987819 210
5                max precision  0.999979 1.000000   0
6                   max recall  0.001865 1.000000 387
7              max specificity  0.999979 1.000000   0
8             max absolute_mcc  0.367137 0.975640 212
9   max min_per_class_accuracy  0.385816 0.987256 204
10 max mean_per_class_accuracy  0.371864 0.987812 210

score 1 · Answer 1 · answered Mar 04 '20 at 16:52

1

In general we would need more info on your data and what you are trying to achieve.

However, with 0.999 AUC, it is very likely you are overfitting (or you have a very simple problem).

answered Mar 04 '20 at 16:52

Lucas Morin

2,513
5
19
39

Hi @lcrmorin, thx for your reply. I thought that 0.999 AUC was good. I was in the false. What is a good value for AUC? I have 7 inputs and i try to train my model with 2 outputs to determine the class Yes or No – user979974 Mar 04 '20 at 17:42

H2O deep learning model performance

1 Answers1