why isn’t numpy.mean multithreaded?

by

I’ve been looking for ways to easily multithread some of my simple analysis code since I had noticed numpy it was only using one core, despite the fact that it is supposed to be multithreaded.

Who says it’s supposed to be multithreaded?

numpy is primarily designed to be as fast as possible on a single core, and to be as parallelizable as possible if you need to do so. But you still have to parallelize it.

In particular, you can operate on independent sub-objects at the same time, and slow operations release the GIL when possible—although “when possible” may not be nearly enough. Also, numpy objects are designed to be shared or passed between processes as easily as possible, to facilitate using multiprocessing.

There are some specialized methods that are automatically parallelized, but most of the core methods are not. In particular, dot is implemented on top of BLAS when possible, and BLAS is automatically parallelized on most platforms, but mean is implemented in plain C code.

See Parallel Programming with numpy and scipy for details.

So, how do you know which methods are parallelized and which aren’t? And, of those which aren’t, how do you know which ones can be nicely manually-threaded and which need multiprocessing?

There’s no good answer to that. You can make educated guesses (X seems like it’s probably implemented on top of ATLAS, and my copy of ATLAS is implicitly threaded), or you can read the source.

But usually, the best thing to do is try it and test. If the code is using 100% of one core and 0% of the others, add manual threading. If it’s now using 100% of one core and 10% of the others and barely running faster, change the multithreading to multiprocessing. (Fortunately, Python makes this pretty easy, especially if you use the Executor classes from concurrent.futures or the Pool classes from multiprocessing. But you still often need to put some thought into it, and test the relative costs of sharing vs. passing if you have large arrays.)

Also, as kwatford points out, just because some method doesn’t seem to be implicitly parallel doesn’t mean it won’t be parallel in the next version of numpy, or the next version of BLAS, or on a different platform, or even on a machine with slightly different stuff installed on it. So, be prepared to re-test. And do something like my_mean = numpy.mean and then use my_mean everywhere, so you can just change one line to my_mean = pool_threaded_mean.

More Related Contents:

  1. Most efficient way to map function over numpy array
  2. Python OpenCV streaming from camera – multithreading, timestamps
  3. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  4. numpy: most efficient frequency counts for unique values in an array
  5. Why is numpy’s einsum faster than numpy’s built in functions?
  6. Limit number of threads in numpy
  7. How to capture multiple camera streams with OpenCV?
  8. Python threads all executing on a single core
  9. Numpy sum elements in array based on its value
  10. Efficiently return the index of the first value satisfying condition in array
  11. Bin elements per row – Vectorized 2D Bincount for NumPy
  12. Vectorized searchsorted numpy
  13. Implement MATLAB’s im2col ‘sliding’ in Python
  14. Frequency counts for unique values in a NumPy array
  15. Most efficient way to forward-fill NaN values in numpy array
  16. Python’s sum vs. NumPy’s numpy.sum
  17. Numpy: Fix array with rows of different lengths by filling the empty elements with zeros
  18. Pandas pd.Series.isin performance with set versus array
  19. Numpy and line intersections
  20. Efficiently detect sign-changes in python
  21. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  22. Efficient way to take the minimum/maximum n values and indices from a matrix using NumPy
  23. Numpy vs Cython speed
  24. numpy float: 10x slower than builtin in arithmetic operations?
  25. Fastest save and load options for a numpy array
  26. No speedup when summing uint16 vs uint64 arrays with NumPy?
  27. Python Numpy Data Types Performance
  28. how to speed up a vector cross product calculation
  29. Getting days since last occurence in Pandas DataFrame?
  30. What is the most efficient way of doing square root of sum of square of two numbers?

Leave a Comment