I have a small dataset of 977 rows with a class proportion of 77:23.
For the sake of metrics improvement, I have kept my minority class ('default') as class 1 (and 'not default' as class 0).
My input variables are categorical (high cardinality) in nature. So, the below is what I tried.
Approach 1
a) Split train and test
b) Apply SMOTE-NC on train data
c) Later, rare_encode and ordinal_encode train and test data separately (due to high cardinal input variables)
d) Build RF model with gridsearch and stratified cross validation, optimizing recall score.
e) Assess the performance
f) No improvement (when compared to imbalanced class. No use of SMOTE NC)
Approach 2
a) Split train and test
b) rare_encode and ordinal_encode train and test data separately (due to high cardinal input variables)
c) SMOTE-NC the train data only
d) Build RF model with gridsearch and stratified cross validation, optimizing recall score.
e) Assess the performance
f) No improvement (when compared to imbalanced class. No use of SMOTE NC)
So, my questions are as follows
a) why there is no improvement of the model especially in test data?
b) Am I doing anything incorrectly with the way am doing SMOTE and encoding of categorical variables?
