Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [boosting]

120 questions
27
votes
1 answer

Adaboost vs Gradient Boosting

How is AdaBoost different from a Gradient Boosting algorithm since both of them use a Boosting technique? I could not figure out actual difference between these both algorithms from a theory point of view.
CodeMaster GoGo
  • 768
  • 1
  • 6
  • 15
12
votes
5 answers

How to make LightGBM to suppress output?

I have tried for a while to figure out how to "shut up" LightGBM. Especially, I would like to suppress the output of LightGBM during training (i.e. feedback on the boosting steps). My model: params = { 'objective': 'regression', …
Peter
  • 7,277
  • 5
  • 18
  • 47
10
votes
4 answers

Can Boosted Trees predict below the minimum value of the training label?

I am using gradient Gradient Boosted Trees (with Catboost) for a Regression task. Can GBtrees predict a label that is below the minimum (or above the max) that was seen in the training ? For instance if the minimum value the label had is 10, would…
Yairh
  • 119
  • 1
  • 5
10
votes
2 answers

What is a good interpretation of this 'learning curve' plot?

I read about the validation_curve and how interpret it to know if there are over-fitting or underfitting, but how can interpret the plot when the data is the error like this: The X-axis is "Nº of examples of training" Redline is train error Green…
Tlaloc-ES
  • 337
  • 1
  • 6
9
votes
1 answer

What is meant by Distributed for a gradient boosting library?

I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library. What is meant by distributed? Have a nice day
Tommaso Bendinelli
  • 275
  • 1
  • 8
9
votes
0 answers

AdaBoost implementation and tuning for high dimensional feature space in R

I am trying to implement the AdaBoost.M1 algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in R. There exists a variety of different packages for this purpose; AdaBag, Ada and gbm.…
AfBM
  • 91
  • 2
7
votes
1 answer

How to extract trees in XGBoost?

I want to extract each tree so that I can feed it with any data, and see the output. dump_list=xg_clas.get_booster().get_dump() num_t=len(dump_list) print("Number of Trees=",num_t) I can find number of trees like this, xgb.plot_tree(xg_clas,…
J.Smith
  • 458
  • 3
  • 15
6
votes
1 answer

Boosting with highly correlated features

I have a conceptual question. My understanding is, that Random Forest can be applied even when features are (highly) correlated. This is because with bagging, the influence of few highly correlated features is moderated, since each feature only…
Peter
  • 7,277
  • 5
  • 18
  • 47
6
votes
1 answer

How can I prevent this model to learn more(less) :)))

As you can see, GradientBoostingClassifier overfit with more training example. These are my parameter for the model: {'learning_rate': 0.1, 'loss': 'deviance', 'max_depth': 6, 'max_features': 0.3, 'min_samples_leaf': 80, 'n_estimators': 300} What…
parvij
  • 761
  • 5
  • 17
6
votes
3 answers

how does XGBoost's exact greedy split finding algorithm determine candidate split values for different feature types?

Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to find the best split" (page 3). Thus, my understanding…
tvl
  • 71
  • 5
5
votes
1 answer

Can Boosting and Bagging be applied to heterogeneous algorithms?

Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in all the iterations, could we use different…
Ahmad Bilal
  • 177
  • 5
5
votes
1 answer

Is the way to combine weak learners in AdaBoost for regression arbitrary?

I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression. In paper Improving Regressors using Boosting Techniques, the final prediction is the weighted median. For a particular…
Akira
  • 175
  • 4
5
votes
1 answer

Extracting encoded features after CatBoost

I have a dataset containing numerical as well as categorical variables. After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables encoded in whatever method the classifier decided to…
Aishwarya A R
  • 229
  • 2
  • 7
5
votes
5 answers

GridSearch without CV

I create a Random Forest and Gradient Boosting Regressor by using GridSearchCV. For the Gradient Boosting Regressor, it takes too long for me. But I need to know which are the best parameters for the models. So I am thinking if there is a GridSearch…
ml_learner
  • 347
  • 1
  • 4
  • 11
5
votes
2 answers

Bagging vs Boosting, Bias vs Variance, Depth of trees

I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees First question: why we are supposed to use weak learners…
K.Hua
  • 153
  • 6
1
2 3 4 5 6 7 8