Getting a low ROC AUC score but a high accuracy

by

To start with, saying that an AUC of 0.583 is “lower” than a score* of 0.867 is exactly like comparing apples with oranges.

[* I assume your score is mean accuracy, but this is not critical for this discussion – it could be anything else in principle]

According to my experience at least, most ML practitioners think that the AUC score measures something different from what it actually does: the common (and unfortunate) use is just like any other the-higher-the-better metric, like accuracy, which may naturally lead to puzzles like the one you express yourself.

The truth is that, roughly speaking, the AUC measures the performance of a binary classifier averaged across all possible decision thresholds.

The (decision) threshold in binary classification is the value above which we decide to label a sample as 1 (recall that probabilistic classifiers actually return a value p in [0, 1], usually interpreted as a probability – in scikit-learn it is what predict_proba returns).

Now, this threshold, in methods like scikit-learn predict which return labels (1/0), is set to 0.5 by default, but this is not the only possibility, and it may not even be desirable in come cases (imbalanced data, for example).

The point to take home is that:

  • when you ask for score (which under the hood uses predict, i.e. labels and not probabilities), you have also implicitly set this threshold to 0.5
  • when you ask for AUC (which, in contrast, uses probabilities returned with predict_proba), no threshold is involved, and you get (something like) the accuracy averaged across all possible thresholds

Given these clarifications, your particular example provides a very interesting case in point:

I get a good-enough accuracy ~ 87% with my model; should I care that, according to an AUC of 0.58, my classifier does only slightly better than mere random guessing?

Provided that the class representation in your data is reasonably balanced, the answer by now should hopefully be obvious: no, you should not care; for all practical cases, what you care for is a classifier deployed with a specific threshold, and what this classifier does in a purely theoretical and abstract situation when averaged across all possible thresholds should pose very little interest for a practitioner (it does pose interest for a researcher coming up with a new algorithm, but I assume that this is not your case).

(For imbalanced data, the argument changes; accuracy here is practically useless, and you should consider precision, recall, and the confusion matrix instead).

For this reason, AUC has started receiving serious criticism in the literature (don’t misread this – the analysis of the ROC curve itself is highly informative and useful); the Wikipedia entry and the references provided therein are highly recommended reading:

Thus, the practical value of the AUC measure has been called into question, raising the possibility that the AUC may actually introduce more uncertainty into machine learning classification accuracy comparisons than resolution.

[…]

One recent explanation of the problem with ROC AUC is that reducing the ROC Curve to a single number ignores the fact that it is about the tradeoffs between the different systems or performance points plotted and not the performance of an individual system

Emphasis mine – see also On the dangers of AUC…

More Related Contents:

  1. Scikit learn – fit_transform on the test set
  2. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  3. Loss & accuracy – Are these reasonable learning curves?
  4. Save classifier to disk in scikit-learn
  5. How to get most informative features for scikit-learn classifiers?
  6. How to interpret MSE in Keras Regressor
  7. What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?
  8. Cost function training target versus accuracy desired goal
  9. Why does one hot encoding improve machine learning performance? [closed]
  10. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  11. How to put more weight on certain features in machine learning?
  12. Cost function in logistic regression gives NaN as a result
  13. How to find the importance of the features for a logistic regression model?
  14. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  15. Early stopping with Keras and sklearn GridSearchCV cross-validation
  16. Scikits-Learn RandomForrest trained on 64bit python wont open on 32bit python
  17. How should I teach machine learning algorithm using data with big disproportion of classes? (SVM)
  18. Under what parameters are SVC and LinearSVC in scikit-learn equivalent?
  19. scikit-learn .predict() default threshold
  20. What is the role of the bias in neural networks? [closed]
  21. Is it possible to specify your own distance function using scikit-learn K-Means Clustering?
  22. Finding K-nearest neighbors and its implementation
  23. Many to one and many to many LSTM examples in Keras
  24. LabelEncoder for categorical features?
  25. Multi-class classification in libsvm [closed]
  26. sklearn Logistic Regression “ValueError: Found array with dim 3. Estimator expected
  27. What is the difference between linear regression and logistic regression? [closed]
  28. Can Keras deal with input images with different size?
  29. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  30. Using Smote with Gridsearchcv in Scikit-learn

Leave a Comment