What are the different use cases of joblib versus pickle?

by
  • joblib is usually significantly faster on large numpy arrays because it has a special handling for the array buffers of the numpy datastructure. To find about the implementation details you can have a look at the source code. It can also compress that data on the fly while pickling using zlib or lz4.
  • joblib also makes it possible to memory map the data buffer of an uncompressed joblib-pickled numpy array when loading it which makes it possible to share memory between processes.
  • if you don’t pickle large numpy arrays, then regular pickle can be significantly faster, especially on large collections of small python objects (e.g. a large dict of str objects) because the pickle module of the standard library is implemented in C while joblib is pure python.
  • since PEP 574 (Pickle protocol 5) has been merged in Python 3.8, it is now much more efficient (memory-wise and cpu-wise) to pickle large numpy arrays using the standard library. Large arrays in this context means 4GB or more.
  • But joblib can still be useful with Python 3.8 to load objects that have nested numpy arrays in memory mapped mode with mmap_mode="r".

More Related Contents:

  1. How can I use pickle to save a dict (or any other Python object)?
  2. Storing Python dictionaries
  3. Can Python pickle lambda functions?
  4. ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
  5. Unpickling a python 2 object with python 3
  6. Accuracy Score ValueError: Can’t Handle mix of binary and continuous target
  7. How to normalize a NumPy array to a unit vector?
  8. Use scikit-learn to classify into multiple categories
  9. sklearn.LabelEncoder with never seen before values
  10. Preserve custom attributes when pickling subclass of numpy array
  11. Custom transformer for sklearn Pipeline that alters both X and y
  12. Scikit Learn SVC decision_function and predict
  13. RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  14. Save MinMaxScaler model in sklearn
  15. sklearn plot confusion matrix with labels
  16. Saving and loading objects and using pickle
  17. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  18. LabelEncoder: TypeError: ‘>’ not supported between instances of ‘float’ and ‘str’
  19. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  20. Run an OLS regression with Pandas Data Frame
  21. How to upgrade scikit-learn package in anaconda
  22. sklearn pipeline – how to apply different transformations on different columns
  23. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  24. Dump a list in a pickle file and retrieve it back later [closed]
  25. Python 3.7 Error: Unsupported Pickle Protocol 5
  26. Python pickle protocol choice?
  27. Understanding Pickling in Python
  28. Pickle with custom classes
  29. Using Smote with Gridsearchcv in Scikit-learn
  30. Python import error: cannot import name ‘six’ from ‘sklearn.externals’

Leave a Comment