Questions tagged [adaboost]
27 questions
9
votes
0 answers
AdaBoost implementation and tuning for high dimensional feature space in R
I am trying to implement the AdaBoost.M1 algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in R. There exists a variety of different packages for this purpose; AdaBag, Ada and gbm.…
AfBM
- 91
- 2
5
votes
1 answer
Is the way to combine weak learners in AdaBoost for regression arbitrary?
I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression.
In paper Improving Regressors using Boosting Techniques, the final prediction is the weighted median.
For a particular…
Akira
- 175
- 4
3
votes
2 answers
How to use a set of pre-defined classifiers in Adaboost?
Suppose there are some classifiers as follows:
dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn =…
Amin
- 191
- 3
- 9
2
votes
1 answer
Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”
The short version: I am trying to compare different classifiers for a certain dataset from kaggle, and am trying to also compare these classifiers between before using PCA (form sklearn) to after using PCA in terms of accuracy and runtime. For some…
appeldaniel
- 41
- 3
2
votes
0 answers
Understanding additive function approximation or Understanding matching pursuit
I am trying to read Greedy function approximation: A gradient boosting machine.
On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach breaks down when we have finite data and some way to…
figs_and_nuts
- 775
- 1
- 4
- 13
2
votes
1 answer
Decreasing n_estimators is increasing accuracy in AdaBoost?
I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset.
(X,Y are the predictor columns and the color is the label)
As you can see there are exactly 16 points in either side that can be easily miss-classified. TO check…
2
votes
0 answers
AdaBoost vs Gradient Boost
What is the difference? Under which criteria should each type of boost be used?
What is the theory behind each of these methods?
Nikita Rogers
- 41
- 1
- 3
1
vote
1 answer
Evaluating optimal values for depth of tree
I'm studying the performance of an AdaBoost model and I wonder how it performs in regard to the depth of the trees.
Here's the accuracy for the model with a depth of 1
and here with a depth of 3
From my point of view, I would say the lower one…
Ben
- 510
- 5
- 13
1
vote
1 answer
How Adaboost calculates error for each weak learner in training?
I am studying the Adaboost classification algorithm because i would like to implement it from scratch. I understand how it works, but i am not able to understand where some steps are placed.
I will describe the Adaboost training steps in my…
heresthebuzz
- 395
- 3
- 9
1
vote
1 answer
Adaboost with other classifier fitting
There is the opportunity to fit decision trees with other decision trees. For example:
adaclassification= AdaBoostClassifier(RandomForestClassifier(n_jobs=-1))
adaclassification.fit(X_train,y_train)
I got better results with random forest, so…
martin
- 329
- 3
- 12
1
vote
2 answers
Does gradient boosting algorithm error always decrease faster and lower on training data?
I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which results in the following loss plot:
I'd say this is…
Xaume
- 182
- 2
- 11
1
vote
0 answers
AdaBoost.R2 learning rate from scikit learn
AdaBoost.R2 (regression), is presented in the paper "improving regressors with boosting techniques" from Drucker and is freely available on Scholar.
The implementation of regression for AdaBoost in scikit learn uses this algorithm (paper is cited in…
Lucien Ledune
- 23
- 2
1
vote
1 answer
Why Adaboost SAMME needs f to be estimable?
I am trying to understand the mathematics behind SAMME AdaBoost:
At some stage, the paper adds a constraint for f to be estimable:
I do not understand why this is required. Can someone explain a bit better why this restriction is needed?
As well,…
Aggamarcel
- 45
- 5
1
vote
1 answer
Explanation on some steps of AdaBoost.R2
I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really get.
The paper is available here, and Adaboost.R2…
Lucien Ledune
- 23
- 2
1
vote
0 answers
AdaBoost decision_function() outputs in binary classification with sklearn
As I understand it based on some study of the source code, I would expect, when using AdaBoost, that values obtained by calling decision_function() would be bounded between -1 and 1. This is because it's the weighed average of the probabilities.…
Steve
- 111
- 2