Loss function for class imbalanced binary classifier in Tensor flow

by

You can add class weights to the loss function, by multiplying logits.
Regular cross entropy loss is this:

loss(x, class) = -log(exp(x[class]) / (\sum_j exp(x[j])))
               = -x[class] + log(\sum_j exp(x[j]))

in weighted case:

loss(x, class) = weights[class] * -x[class] + log(\sum_j exp(weights[class] * x[j]))

So by multiplying logits, you are re-scaling predictions of each class by its class weight.

For example:

ratio = 31.0 / (500.0 + 31.0)
class_weight = tf.constant([ratio, 1.0 - ratio])
logits = ... # shape [batch_size, 2]
weighted_logits = tf.mul(logits, class_weight) # shape [batch_size, 2]
xent = tf.nn.softmax_cross_entropy_with_logits(
  weighted_logits, labels, name="xent_raw")

There is a standard losses function now that supports weights per batch:

tf.losses.sparse_softmax_cross_entropy(labels=label, logits=logits, weights=weights)

Where weights should be transformed from class weights to a weight per example (with shape [batch_size]). See documentation here.

More Related Contents:

  1. FailedPreconditionError: Attempting to use uninitialized in Tensorflow
  2. What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?
  3. How to compile Tensorflow with SSE4.2 and AVX instructions?
  4. What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits?
  5. Keras + Tensorflow and Multiprocessing in Python
  6. How to get most informative features for scikit-learn classifiers?
  7. Is Tensorflow compatible with a Windows workflow?
  8. Predict classes or class probabilities?
  9. Getting a low ROC AUC score but a high accuracy
  10. Tensorflow, best way to save state in RNNs?
  11. Remove nodes from graph or reset entire default graph
  12. How does Keras 1d convolution layer work with word embeddings – text classification problem? (Filters, kernel size, and all hyperparameter)
  13. What are the differences between all these cross-entropy losses in Keras and TensorFlow?
  14. What do I need K.clear_session() and del model for (Keras with Tensorflow-gpu)?
  15. Tensorflow crashes with CUBLAS_STATUS_ALLOC_FAILED
  16. A simple explanation of Naive Bayes Classification [closed]
  17. Tensorflow: Using Adam optimizer
  18. Tensorflow: executing an ops with a specific core of a CPU
  19. How to convert .pb to TFLite format?
  20. Load Tensorflow js model from local file system in javascript
  21. How to select rows from a 3-D Tensor in TensorFlow?
  22. memory issues when transforming np.array using to_categorical
  23. Working with multiple graphs in TensorFlow
  24. “freeze” some variables/scopes in tensorflow: stop_gradient vs passing variables to minimize
  25. overcome Graphdef cannot be larger than 2GB in tensorflow
  26. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  27. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  28. What is right batch normalization function in Tensorflow?
  29. Strange behaviour of the loss function in keras model, with pretrained convolutional base
  30. Checkpointing keras model: TypeError: can’t pickle _thread.lock objects

Leave a Comment