How does sklearn.svm.svc’s function predict_proba() work internally?

by

Scikit-learn uses LibSVM internally, and this in turn uses Platt scaling, as detailed in this note by the LibSVM authors, to calibrate the SVM to produce probabilities in addition to class predictions.

Platt scaling requires first training the SVM as usual, then optimizing parameter vectors A and B such that

P(y|X) = 1 / (1 + exp(A * f(X) + B))

where f(X) is the signed distance of a sample from the hyperplane (scikit-learn’s decision_function method). You may recognize the logistic sigmoid in this definition, the same function that logistic regression and neural nets use for turning decision functions into probability estimates.

Mind you: the B parameter, the “intercept” or “bias” or whatever you like to call it, can cause predictions based on probability estimates from this model to be inconsistent with the ones you get from the SVM decision function f. E.g. suppose that f(X) = 10, then the prediction for X is positive; but if B = -9.9 and A = 1, then P(y|X) = .475. I’m pulling these numbers out of thin air, but you’ve noticed that this can occur in practice.

Effectively, Platt scaling trains a probability model on top of the SVM’s outputs under a cross-entropy loss function. To prevent this model from overfitting, it uses an internal five-fold cross validation, meaning that training SVMs with probability=True can be quite a lot more expensive than a vanilla, non-probabilistic SVM.

More Related Contents:

  1. Scikit Learn SVC decision_function and predict
  2. Does the SVM in sklearn support incremental (online) learning?
  3. Determining the most contributing features for SVM classifier in sklearn
  4. How to extract the decision rules from scikit-learn decision-tree?
  5. Is it possible to specify your own distance function using scikit-learn K-Means Clustering?
  6. Find p-value (significance) in scikit-learn LinearRegression
  7. Stratified Train/Test-split in scikit-learn
  8. Error importing scikit-learn modules
  9. Efficiently create sparse pivot tables in pandas?
  10. LabelEncoder for categorical features?
  11. Impute categorical missing values in scikit-learn
  12. Scikit-learn’s LabelBinarizer vs. OneHotEncoder
  13. Should I use `random.seed` or `numpy.random.seed` to control random number generation in `scikit-learn`?
  14. Can You Consistently Keep Track of Column Labels Using Sklearn’s Transformer API?
  15. Passing categorical data to Sklearn Decision Tree
  16. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  17. How does the class_weight parameter in scikit-learn work?
  18. Fastest pairwise distance metric in python
  19. Getting model attributes from pipeline
  20. Efficiently count word frequencies in python
  21. sklearn Logistic Regression “ValueError: Found array with dim 3. Estimator expected
  22. Sklearn Pipeline: Get feature names after OneHotEncode In ColumnTransformer
  23. How to find the importance of the features for a logistic regression model?
  24. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  25. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  26. sklearn : TFIDF Transformer : How to get tf-idf values of given words in document
  27. Using scikit to determine contributions of each feature to a specific class prediction
  28. In sklearn.decomposition.PCA, why are components_ negative?
  29. Pyinstaller ; ModuleNotFoundError: No module named ‘sklearn.utils._cython_blas’
  30. Model help using Scikit-learn when using GridSearch

Leave a Comment