2

I am solving a Kaggle contest and my single model has reached score of 0.121, I'd like to know when to start using ensembling/stacking to improve the score.

I used lasso and xgboost and there obviously must be variance associated with those two algorithms. So stacking should theoretically give me better output than my individual algorithms.

But how to idenfity if stacking is worth it and that we've reached dead end to accuracy of a particular model?

thewhitetulip
  • 153
  • 1
  • 6

1 Answers1

1

Stacking is going to help most when individual models capture unique characteristics of the data. It is often the case that different architectures perform similarly, if somewhat differently, on the same data. In those cases, ensembling/stacking will only offer slight incremental benefits. In the limit, of you only care about prediction, you can wire up as many different approaches as you can think of. However if interpretability is key, each additional component model will further complicate things.

Your specific question of when to know if it’s worth it or if you’ve reached the limit can be treated like anything else - is your incremental r-square/error/classification accuracy significantly better versus a simpler approach?

HEITZ
  • 911
  • 4
  • 7
  • Yes, I get that. But my question is a little different. The thing is, I am doing the toy contest on Kaggle and there people have crazy scores like RMSE of 0 and 0.02, my best public score has been 0.12. Since I am a newbie to ML, I am not understanding when to stop trying to tune a model. Even when my rmse was 0.14 I was like 'this is the best I can do' and a month later I got it down to 0.12 by tweaking the features. That's what I wanted to know, is there a statistical way to identify when to stop tuning a single model or do I try stacking and figure that out by doing it. – thewhitetulip Dec 05 '19 at 02:35
  • There are statistical tests for change in, say R-square for example. There are also rules of thumb for convergence criteria that determine when to stop trying. But it’s a judgement call. On the other end of the spectrum you could loop from 1 to infinity each time changing your random seed hoping to capitalize on chance. – HEITZ Dec 05 '19 at 04:11
  • Got it. Thank you. – thewhitetulip Dec 06 '19 at 03:29