How to find the importance of the features for a logistic regression model?

by

One of the simplest options to get a feeling for the “influence” of a given parameter in a linear classification model (logistic being one of those), is to consider the magnitude of its coefficient times the standard deviation of the corresponding parameter in the data.

Consider this example:

import numpy as np    
from sklearn.linear_model import LogisticRegression

x1 = np.random.randn(100)
x2 = 4*np.random.randn(100)
x3 = 0.5*np.random.randn(100)
y = (3 + x1 + x2 + x3 + 0.2*np.random.randn()) > 0
X = np.column_stack([x1, x2, x3])

m = LogisticRegression()
m.fit(X, y)

# The estimated coefficients will all be around 1:
print(m.coef_)

# Those values, however, will show that the second parameter
# is more influential
print(np.std(X, 0)*m.coef_)

An alternative way to get a similar result is to examine the coefficients of the model fit on standardized parameters:

m.fit(X / np.std(X, 0), y)
print(m.coef_)

Note that this is the most basic approach and a number of other techniques for finding feature importance or parameter influence exist (using p-values, bootstrap scores, various “discriminative indices”, etc).

I am pretty sure you would get more interesting answers at https://stats.stackexchange.com/.

More Related Contents:

  1. How to extract the decision rules from scikit-learn decision-tree?
  2. scikit-learn & statsmodels – which R-squared is correct?
  3. Save classifier to disk in scikit-learn
  4. How to implement the Softmax function in Python
  5. Accuracy Score ValueError: Can’t Handle mix of binary and continuous target
  6. Is it possible to specify your own distance function using scikit-learn K-Means Clustering?
  7. How to get most informative features for scikit-learn classifiers?
  8. How to get precision, recall and f-measure from confusion matrix in Python [duplicate]
  9. LabelEncoder for categorical features?
  10. Custom transformer for sklearn Pipeline that alters both X and y
  11. Save MinMaxScaler model in sklearn
  12. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  13. sklearn Logistic Regression “ValueError: Found array with dim 3. Estimator expected
  14. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  15. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  16. Can anyone explain me StandardScaler?
  17. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  18. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  19. scikit-learn random state in splitting dataset
  20. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  21. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  22. scikit-learn: how to scale back the ‘y’ predicted result
  23. Tensorflow Precision / Recall / F1 score and Confusion matrix
  24. Using Smote with Gridsearchcv in Scikit-learn
  25. confused about random_state in decision tree of scikit learn
  26. Does GridSearchCV perform cross-validation?
  27. Can I send callbacks to a KerasClassifier?
  28. How to get most informative features for scikit-learn classifier for different class?
  29. Does the SVM in sklearn support incremental (online) learning?
  30. scikit-learn .predict() default threshold

Leave a Comment