How are feature_importances in RandomForestClassifier determined?

by

There are indeed several ways to get feature “importances”. As often, there is no strict consensus about what this word means.

In scikit-learn, we implement the importance as described in [1] (often cited, but unfortunately rarely read…). It is sometimes called “gini importance” or “mean decrease impurity” and is defined as the total decrease in node impurity (weighted by the probability of reaching that node (which is approximated by the proportion of samples reaching that node)) averaged over all trees of the ensemble.

In the literature or in some other packages, you can also find feature importances implemented as the “mean decrease accuracy”. Basically, the idea is to measure the decrease in accuracy on OOB data when you randomly permute the values for that feature. If the decrease is low, then the feature is not important, and vice-versa.

(Note that both algorithms are available in the randomForest R package.)

[1]: Breiman, Friedman, “Classification and regression trees”, 1984.

More Related Contents:

  1. Why does the following code cant get value of n_fold = 1?
  2. How to extract the decision rules from scikit-learn decision-tree?
  3. sklearn: Found arrays with inconsistent numbers of samples when calling LinearRegression.fit()
  4. Why is Random Forest with a single tree much better than a Decision Tree classifier?
  5. Feature/Variable importance after a PCA analysis
  6. What does `sample_weight` do to the way a `DecisionTreeClassifier` works in sklearn?
  7. Got continuous is not supported error in RandomForestRegressor
  8. Scikit learn – fit_transform on the test set
  9. python kmeans on string
  10. How to one-hot-encode from a pandas column containing a list?
  11. scikit-learn & statsmodels – which R-squared is correct?
  12. ImportError in importing from sklearn: cannot import name check_build
  13. Random state (Pseudo-random number) in Scikit learn
  14. Error importing scikit-learn modules
  15. How to one hot encode variant length features?
  16. Clustering text documents using scikit-learn kmeans in Python
  17. scikit-learn cross validation, negative values with mean squared error
  18. Efficiently create sparse pivot tables in pandas?
  19. ROC for multiclass classification
  20. fit-transform on training data and transform on test data [duplicate]
  21. Why is pydot unable to find GraphViz’s executables in Windows 8?
  22. How to find the importance of the features for a logistic regression model?
  23. scikit-learn random state in splitting dataset
  24. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  25. sklearn : TFIDF Transformer : How to get tf-idf values of given words in document
  26. Early stopping with Keras and sklearn GridSearchCV cross-validation
  27. Numpy Array Get row index searching by a row
  28. Using scikit to determine contributions of each feature to a specific class prediction
  29. Model help using Scikit-learn when using GridSearch
  30. Determining the most contributing features for SVM classifier in sklearn

Leave a Comment