Parallelizing a Numpy vector operation

by

There is a better way: numexpr

Slightly reworded from their main page:

It’s a multi-threaded VM written in C that analyzes expressions, rewrites them more efficiently, and compiles them on the fly into code that gets near optimal parallel performance for both memory and cpu bounded operations.

For example, in my 4 core machine, evaluating a sine is just slightly less than 4 times faster than numpy.

In [1]: import numpy as np
In [2]: import numexpr as ne
In [3]: a = np.arange(1000000)
In [4]: timeit ne.evaluate('sin(a)')
100 loops, best of 3: 15.6 ms per loop    
In [5]: timeit np.sin(a)
10 loops, best of 3: 54 ms per loop

Documentation, including supported functions here. You’ll have to check or give us more information to see if your more complicated function can be evaluated by numexpr.

More Related Contents:

  1. Use numpy array in shared memory for multiprocessing
  2. Shared-memory objects in multiprocessing
  3. Why does multiprocessing use only a single core after I import numpy?
  4. Is shared readonly data copied to different processes for multiprocessing?
  5. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  6. Multiprocessing.Pool makes Numpy matrix multiplication slower
  7. How to share numpy random state of a parent process with child processes?
  8. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  9. Conditionally fill column values based on another columns value in pandas
  10. How to convert an array of strings to an array of floats in numpy?
  11. How do I convert a PIL Image into a NumPy array?
  12. How do I remove NaN values from a NumPy array?
  13. NumPy 1.21.2 may not yet support Python 3.10
  14. “In” operator for numpy arrays?
  15. Problem in combining bar plot and line plot (python)
  16. yet another confusion with multiprocessing error, ‘module’ object has no attribute ‘f’
  17. PIP Install Numpy throws an error “ascii codec can’t decode byte 0xe2”
  18. vectorize conditional assignment in pandas dataframe
  19. Efficient way to partially read large numpy file?
  20. Multiprocessing launching too many instances of Python VM
  21. Convert numpy type to python
  22. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  23. Screen Capture with OpenCV and Python-2.7
  24. Find the index of the k smallest values of a numpy array
  25. Principal component analysis in Python
  26. Numpy: use reshape or newaxis to add dimensions
  27. Stratified Sampling in Pandas
  28. pylab.hist(data, normed=1). Normalization seems to work incorrect
  29. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  30. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?

Leave a Comment