Somehow I missed this question... I can answer this as I have spent a great part of the last years trying to deal with imbalance inside and outside of Kaggle.
Why SMOTE doesn't work?
SMOTE creates syntectic data points, which is roughly equivalent to oversampling (if you take points A and B, create (A+B)/2, it is roughly equivalent to weighting A and B by 1.5). The added complexity of the generation process doesn't seems to bring consistent performance.
I even think there might be some sort of 'no free lunch theorem' here: as you don't add or remove information, sometimes the new points are helpful, sometimes they are counter-productive, and the average impact should be 0.
Now the question is: does oversampling work ? Well it seems we have a decent answer over here: https://stats.stackexchange.com/questions/357466/are-unbalanced-datasets-problematic-and-how-does-oversampling-purport-to-he (roughly: No). I think that generally answer the question of the usefullness of SMOTE.
There are some times where undersampling/oversampling is actually needed for engineering constraints (undersampling reduces data size, and hence memory usage; oversampling ensures you have positives exemples in each mini-batches for NN, and it seems to accelerate convergence)
Why is it popular despite not being used on Kaggle?
Regarding Kaggle, I remember reading a winning solution mentioning SMOTE and that it only has marginal effect. I can't manage to find the discussion post. It might have been a minor / custom comp. I recently checked the blog posts and there is indeed no mention of SMOTE. But, why has it gained so much traction despite not being effective on Kaggle ?
- Bad research practices (a.k.a. 'publish or perish') that only lead to diffusion of apparent increment on this solution, marginally new techniques, non-robust increment in performance ... etc. despite any application on any real life data set. Some of those practices can be found in business too.
- Bad influencer practices (you know those linkedin people sharing half baked Tds article), that are led by visibility instead of quality. Unfortunately, this sort of behavior is also present on Kaggle forums (enven encouraged by the medal system).
- Bad interviewing practices: somehow it has become an interview question, it appears on interview question lists. Now you have both non-technical people asking about it and young DS learning to answer 'SMOTE' when they are asked about data imbalance. A context were truth matters less than confidence.
- Bad ML practices: Using wrong metrics and poorly designed cv it is very easy to gain performance as SMOTE is very leaky (it get easier to predict B from A when you add (A+B)/2 in your fold). It is then easier to publish with a gap up in performance and put another coin in the hype machine.
- Bad evaluation practice: even with a good metric if you add a bit of noise in your data you have roughly 50% chance to improve the performance. It is often really easy (often incentivised) to be lead by non-robust evaluation, both inside and outside of Kaggle (business and research). It is quite rare to see discussion of robust evaluation of performance increment, even on kaggle.
Last but not least: Kagglers know a way better alternative. It should not be a surprise but Kaggle is very into gbdt that appears to be quite effective at handling imbalance. Notably each leaf of each tree being a set of exemple, you usually have a good probability calibration. A little bit of regularisation and you are good to go.
One major caveat to all of this: tabular comp. seems to get rarer over time (outside of TPS), and (maybe because they figured it is not a problem) there wasn't a significant imbalanced one in a while.
Cherry on top: you might want to see this Euroscipy Video from last week. G. Lemaitre from sklearn shows how to use it on an imbalanced dataset. It includes a demonstration of how SMOTE doesn't work. He also mention the package Imbalanced-Learn, that, as one of the creators he advise not to use (putting ADASYN and Tomek links in the same garbage bin). He also ends up talking about the real problem: cost imbalance matters more for business than target imbalance.
