The goal of Ensemble methods is to combine the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator.OR Ensemble methods is a machine learning technique that combines several base models in order to produce one optimal predictive OR Ensemble is the art of combining diverse set of learners (individual models) together to improvise on the stability and predictive power of the model

1. averaging methods, The driving principle is to build several estimators independently and then to average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced. Examples: Bagging methods, Forests of randomized trees, …

2. boosting methods, Base estimators are built sequentially and one tries to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble. Examples: AdaBoost, Gradient Tree Boosting

1. Bagging - Building multiple models (typically of the same type) from different subsamples of the training dataset.

2. Boosting - Building multiple models (typically of the same type) each of which learns to fix the prediction errors of a prior model in the chain.

3. Voting - Building multiple models (typically of differing types) and simple statistics (like calculating the mean) are used to combine predictions.

4. Bagging Algorithms Bootstrap Aggregation or bagging involves taking multiple samples from your training dataset (with replacement) and training a model for each sample. The final output prediction is averaged across the predictions of all of the sub-models.

(a). Bagged Decision Trees Bagging performs best with algorithms that have high variance. A popular example are decision trees, often constructed without pruning

(b). Random Forest Samples of the training dataset are taken with replacement, but the trees are constructed in a way that reduces the correlation between individual classifiers. Specifically, rather than greedily choosing the best split point in the construction of the tree, only a random subset of features are considered for each split.

(c).Extra Trees Extra Trees are another modification of bagging where random trees are constructed from samples of the training dataset.

2. Boosting Algorithm Boosting ensemble algorithms creates a sequence of models that attempt to correct the mistakes of the models before them in the sequence.Once created, the models make predictions which may be weighted by their demonstrated accuracy and the results are combined to create a final output prediction.

(a). AdaBoost AdaBoost was perhaps the first successful boosting ensemble algorithm. It generally works by weighting instances in the dataset by how easy or difficult they are to classify, allowing the algorithm to pay more or less attention to them in the construction of subsequent models.

(b). Stochastic Gradient Boosting(the best) Stochastic Gradient Boosting (also called Gradient Boosting Machines) are one of the most sophisticated ensemble techniques. It is also a technique that is proving to be perhaps of the the best techniques available for improving performance via ensembles

RESOURCES: 1.https://www.analyticsvidhya.com/blog/2015/08/introduction-ensemble-learning/