donny-hikari / adaboost Goto Github PK

AdaBoost is short for Adaptive Boosting. Boosting is one method of Ensemble Learning. There are other Ensemble Learning methods like Bagging, Stacking, etc.. The differences between Bagging, Boosting, Stacking are:

1. Bagging:
   Equal weight voting. Trains each model with a random drawn subset of training set.

2. Boosting:
   Trains each new model instance to emphasize the training instances that previous models mis-classified. Has better accuracy comparing to bagging, but also tends to overfit.

3. Stacking:
   Trains a learning algorithm to combine the predictions of several other learning algorithms.

Given a N*M matrix X, and a N vector y, where N is the count of samples, and M is the features of samples. AdaBoost trains T weak classifiers with the following steps:

1. Initialize.

2. Train i-th weak classifier with training set {X, y} and distribution W_i.

3. Get the predict result \( h_i \) on the weak classifier with input X.

4. Update.

Z is a normalization factor.

5. Repeat steps 2 ~ 4 until i reaches T.

6. Output the final hypothesis:

Using a demo from sklearn AdaBoost, I got the following result.

Weak classifiers: 200; Iteration steps in each weak classifier: 200:

Weak classifiers: 60; Iteration steps in each weak classifier: 60:

Weak classifiers: 400; Iteration steps in each weak classifier: 400:

We can see the result varies as the number of weak classifiers and the iteration steps change.

Compares with the AdaBoostClassifier from sklearn with 200 estimators (weak classifiers):

More comparison for my AdaBoostClassifier with different parameters:

1. AdaBoost - Wikipedia
2. AdaBoost -- 从原理到实现
3. A Short Introduction to Boosting
4. Multi-class AdaBoost

Donny

Find out further information, refer to my blog AdaBoost - Donny (In Chinese & English).