I tried to use monotonic constraints in LGBM, but if I use mean absolute error as an objective, it gives a warning that monotonic constraints cannot be done in l1.
What is the reason? Thanks!
Asked
Active
Viewed 127 times
1
Ben Reiniger
- 11,094
- 3
- 16
- 53
morqueatsz
- 25
- 3
-
Could you share the error traceback, so we can see where in the code the warning is produced? – Ben Reiniger Jan 23 '23 at 14:36
1 Answers
2
MAE is an odd loss function for GBMs: the gradient is constant ($\pm1$), and the hessian all zeros, so the usual tree-training target of $-G/H$ (possibly with additional terms for regularization) doesn't work.
Monotone constraints are implemented by rejecting splits that would produce node values (a.k.a. "weights" or "scores", the average target value among observations at that node) that would violate the specified directionality; see this blog post for details.
In LightGBM, MAE is handled by overriding the leaf scores at the end (see this comment on their github). But that means those adjusted scores are not available during tree construction, and so the monotone constraint requirements cannot be checked. They could probably add this alternative score calculation throughout; I'm not sure offhand if it would preserve monotonicity, but it even if so, it would be some effort for the PR, and increased training time.
Ben Reiniger
- 11,094
- 3
- 16
- 53
-
1Thanks! It seems like Hubert loss can be used with monotonic constraint – morqueatsz Jan 25 '23 at 00:39