4

in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be able to compare the coefficients of each regression model between each other. Example: I have

  • Lasso
  • Ridge
  • ElasticNet
  • AdaBoostRegressor
  • GradientBoostRegressor

I am aware that I could get the coefficients of Lasso, Ridge, and ElasticNet from model.coef_ and model.intercept_ from sklearn. However, AdaBoostRegressor does not have this, but rather, it has weights assigned to each of the estimators. Similarly, GradientBoostingRegressor of sklearn does not even have weights at all. How am I to compare the coefficients between each of the regressive models I am using??

Perl Del Rey
  • 187
  • 5
  • Why do you want to do this? By definition different models work in different ways: assigning coefficients is not the only way and even for those which do the coefficients might not be comparable. – Erwan Jan 26 '20 at 14:13
  • @Erwan I am interested to see the weights each model will assign each feature I have in my data, and see areas where the models assign different weights to some feature Xi for example, while they might all agree on assigning low weight for example to some feature Xj – Perl Del Rey Jan 26 '20 at 15:56

2 Answers2

3

Weights across different types of models are not always comparable, so I think that it would make more sense to do this kind of comparison not across different types of model but within a single type of model varying:

  • the hyper-parameters (if any),
  • the set of instances (e.g. selecting different subsets randomly),
  • the set of features.

I'd recommend in particular varying the instances: if the weight of a feature tends to change a lot depending on the training set it's a sign of overfitting.

If the goal is to compare the importance of some specific features for different models, it's probably more reliable to directly evaluate how the model performs with/without the feature(s).

Erwan
  • 24,823
  • 3
  • 13
  • 34
3

It's not actually possible to directly compare model coefficients. What you might do that would make more sense is to compare similar metrics.

A good start would be to learn about explainability metrics that are comparable across models : LIME, SHAP... etc. (see here : https://christophm.github.io/interpretable-ml-book/) to see how models reacts on different features.

Lucas Morin
  • 2,513
  • 5
  • 19
  • 39