How should I teach machine learning algorithm using data with big disproportion of classes? (SVM)

by

The most basic approach here is to use so called “class weighting scheme” – in classical SVM formulation there is a C parameter used to control the missclassification count. It can be changed into C1 and C2 parameters used for class 1 and 2 respectively. The most common choice of C1 and C2 for a given C is to put

C1 = C / n1
C2 = C / n2

where n1 and n2 are sizes of class 1 and 2 respectively. So you “punish” SVM for missclassifing the less frequent class much harder then for missclassification the most common one.

Many existing libraries (like libSVM) supports this mechanism with class_weight parameters.

Example using python and sklearn

print __doc__

import numpy as np
import pylab as pl
from sklearn import svm

# we create 40 separable points
rng = np.random.RandomState(0)
n_samples_1 = 1000
n_samples_2 = 100
X = np.r_[1.5 * rng.randn(n_samples_1, 2),
          0.5 * rng.randn(n_samples_2, 2) + [2, 2]]
y = [0] * (n_samples_1) + [1] * (n_samples_2)

# fit the model and get the separating hyperplane
clf = svm.SVC(kernel="linear", C=1.0)
clf.fit(X, y)

w = clf.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - clf.intercept_[0] / w[1]


# get the separating hyperplane using weighted classes
wclf = svm.SVC(kernel="linear", class_weight={1: 10})
wclf.fit(X, y)

ww = wclf.coef_[0]
wa = -ww[0] / ww[1]
wyy = wa * xx - wclf.intercept_[0] / ww[1]

# plot separating hyperplanes and samples
h0 = pl.plot(xx, yy, 'k-', label="no weights")
h1 = pl.plot(xx, wyy, 'k--', label="with weights")
pl.scatter(X[:, 0], X[:, 1], c=y, cmap=pl.cm.Paired)
pl.legend()

pl.axis('tight')
pl.show()

In particular, in sklearn you can simply turn on the automatic weighting by setting class_weight="auto".

Visualization of above code from sklearn documentation

More Related Contents:

  1. Under what parameters are SVC and LinearSVC in scikit-learn equivalent?
  2. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  3. How to interpret MSE in Keras Regressor
  4. Getting a low ROC AUC score but a high accuracy
  5. Why does one hot encoding improve machine learning performance? [closed]
  6. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  7. How to put more weight on certain features in machine learning?
  8. What is the difference between supervised learning and unsupervised learning? [closed]
  9. Early stopping with Keras and sklearn GridSearchCV cross-validation
  10. Scikits-Learn RandomForrest trained on 64bit python wont open on 32bit python
  11. Scikit learn – fit_transform on the test set
  12. Does the SVM in sklearn support incremental (online) learning?
  13. Determining the most contributing features for SVM classifier in sklearn
  14. Machine understand natural language – nlp [closed]
  15. Speeker Identification (Machine Learning)
  16. How to extract the decision rules from scikit-learn decision-tree?
  17. scikit-learn & statsmodels – which R-squared is correct?
  18. Is there a rule-of-thumb for how to divide a dataset into training and validation sets? [closed]
  19. What is the difference between steps and epochs in TensorFlow?
  20. What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?
  21. Order between using validation, training and test sets
  22. TensorBoard – Plot training and validation losses on the same graph?
  23. How to create a new gym environment in OpenAI?
  24. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  25. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  26. using precomputed kernels with libsvm
  27. Caffe: What can I do if only a small batch fits into memory?
  28. 10 fold cross-validation in one-against-all SVM (using LibSVM)
  29. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  30. scikit-learn: how to scale back the ‘y’ predicted result

Leave a Comment