Why does multiprocessing use only a single core after I import numpy?

by

After some more googling I found the answer here.

It turns out that certain Python modules (numpy, scipy, tables, pandas, skimage…) mess with core affinity on import. As far as I can tell, this problem seems to be specifically caused by them linking against multithreaded OpenBLAS libraries.

A workaround is to reset the task affinity using

os.system("taskset -p 0xff %d" % os.getpid())

With this line pasted in after the module imports, my example now runs on all cores:

htop_workaround

My experience so far has been that this doesn’t seem to have any negative effect on numpy‘s performance, although this is probably machine- and task-specific .

Update:

There are also two ways to disable the CPU affinity-resetting behaviour of OpenBLAS itself. At run-time you can use the environment variable OPENBLAS_MAIN_FREE (or GOTOBLAS_MAIN_FREE), for example

OPENBLAS_MAIN_FREE=1 python myscript.py

Or alternatively, if you’re compiling OpenBLAS from source you can permanently disable it at build-time by editing the Makefile.rule to contain the line

NO_AFFINITY=1

More Related Contents:

  1. Use numpy array in shared memory for multiprocessing
  2. Shared-memory objects in multiprocessing
  3. Is shared readonly data copied to different processes for multiprocessing?
  4. Compiling numpy with OpenBLAS integration
  5. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  6. Link ATLAS/MKL to an installed Numpy
  7. How to check BLAS/LAPACK linkage in NumPy and SciPy?
  8. multithreaded blas in python/numpy
  9. calling dot products and linear algebra operations in Cython?
  10. Multiprocessing.Pool makes Numpy matrix multiplication slower
  11. Parallelizing a Numpy vector operation
  12. Numpy Pure Functions for performance, caching
  13. Find out if/which BLAS library is used by Numpy
  14. OSError 38 [Errno 38] with multiprocessing
  15. How to share numpy random state of a parent process with child processes?
  16. Python multiprocessing memory usage
  17. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  18. Python multiprocessing linux windows difference
  19. Simple Digit Recognition OCR in OpenCV-Python
  20. How to find the groups of consecutive elements in a NumPy array
  21. How to understand numpy strides for layman?
  22. Installing Numpy on 64bit Windows 7 with Python 2.7.3 [closed]
  23. How do I print the full NumPy array, without truncation?
  24. Increment Numpy array with repeated indices
  25. numpy.array.__iadd__ and repeated indices [duplicate]
  26. Polar contour plot in matplotlib – best (modern) way to do it?
  27. Why is a `for` over a Python list faster than over a Numpy array?
  28. Is there a numpy builtin to reject outliers from a list
  29. How to determine whether a column/variable is numeric or not in Pandas/NumPy?
  30. numpy arange: how to make “precise” array of floats?

Leave a Comment