Why is Random Forest with a single tree much better than a Decision Tree classifier?

by

The random forest estimators with one estimator isn’t just a decision tree?

Well, this is a good question, and the answer turns out to be no; the Random Forest algorithm is more than a simple bag of individually-grown decision trees.

Apart from the randomness induced from ensembling many trees, the Random Forest (RF) algorithm also incorporates randomness when building individual trees in two distinct ways, none of which is present in the simple Decision Tree (DT) algorithm.

The first is the number of features to consider when looking for the best split at each tree node: while DT considers all the features, RF considers a random subset of them, of size equal to the parameter max_features (see the docs).

The second is that, while DT considers the whole training set, a single RF tree considers only a bootstrapped sub-sample of it; from the docs again:

The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if bootstrap=True (default).

The RF algorihm is essentially the combination of two independent ideas: bagging, and random selection of features (see the Wikipedia entry for a nice overview). Bagging is essentially my second point above, but applied to an ensemble; random selection of features is my first point above, and it seems that it had been independently proposed by Tin Kam Ho before Breiman’s RF (again, see the Wikipedia entry). Ho had already suggested that random feature selection alone improves performance. This is not exactly what you have done here (you still use the bootstrap sampling idea from bagging, too), but you could easily replicate Ho’s idea by setting bootstrap=False in your RandomForestClassifier() arguments. The fact is that, given this research, the difference in performance is not unexpected…

To replicate exactly the behaviour of a single tree in RandomForestClassifier(), you should use both bootstrap=False and max_features=None arguments, i.e.

clf = RandomForestClassifier(n_estimators=1, max_features=None, bootstrap=False)

in which case neither bootstrap sampling nor random feature selection will take place, and the performance should be roughly equal to that of a single decision tree.

More Related Contents:

  1. How to extract the decision rules from scikit-learn decision-tree?
  2. confused about random_state in decision tree of scikit learn
  3. Will it lead to Overfitting / Curse of Dimensionality
  4. Why does the following code cant get value of n_fold = 1?
  5. scikit-learn & statsmodels – which R-squared is correct?
  6. Is it possible to specify your own distance function using scikit-learn K-Means Clustering?
  7. How to get most informative features for scikit-learn classifiers?
  8. Predict classes or class probabilities?
  9. Feature/Variable importance after a PCA analysis
  10. Recovering features names of explained_variance_ratio_ in PCA with sklearn
  11. LabelEncoder for categorical features?
  12. What is exactly sklearn.pipeline.Pipeline?
  13. difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
  14. Passing categorical data to Sklearn Decision Tree
  15. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  16. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  17. Can anyone explain me StandardScaler?
  18. How to find the importance of the features for a logistic regression model?
  19. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  20. scikit-learn random state in splitting dataset
  21. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  22. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  23. Tensorflow Precision / Recall / F1 score and Confusion matrix
  24. Got continuous is not supported error in RandomForestRegressor
  25. Can I send callbacks to a KerasClassifier?
  26. How to get most informative features for scikit-learn classifier for different class?
  27. Model help using Scikit-learn when using GridSearch
  28. Does the SVM in sklearn support incremental (online) learning?
  29. Determining the most contributing features for SVM classifier in sklearn
  30. scikit-learn .predict() default threshold

Leave a Comment