Multiprocessing.Pool makes Numpy matrix multiplication slower

by

Regarding the fact that all of your processes are running on the same CPU, see my answer here.

During import, numpy changes the CPU affinity of the parent process, such that when you later use Pool all of the worker processes that it spawns will end up vying for for the same core, rather than using all of the cores available on your machine.

You can call taskset after you import numpy to reset the CPU affinity so that all cores are used:

import numpy as np
import os
from timeit import timeit
from multiprocessing import Pool


def mmul(matrix):
    for i in range(100):
        matrix = matrix * matrix
    return matrix

if __name__ == '__main__':

    matrices = []
    for i in range(4):
        matrices.append(np.random.random_integers(100, size=(1000, 1000)))

    print timeit(lambda: map(mmul, matrices), number=20)

    # after importing numpy, reset the CPU affinity of the parent process so
    # that it will use all cores
    os.system("taskset -p 0xff %d" % os.getpid())

    pool = Pool(8)
    print timeit(lambda: pool.map(mmul, matrices), number=20)

Output:

    $ python tmp.py                                     
    12.4765810966
    pid 29150's current affinity mask: 1
    pid 29150's new affinity mask: ff
    13.4136221409

If you watch CPU useage using top while you run this script, you should see it using all of your cores when it executes the ‘parallel’ part. As others have pointed out, in your original example the overhead involved in pickling data, process creation etc. probably outweigh any possible benefit from parallelisation.

Edit: I suspect that part of the reason why the single process seems to be consistently faster is that numpy may have some tricks for speeding up that element-wise matrix multiplication that it cannot use when the jobs are spread across multiple cores.

For example, if I just use ordinary Python lists to compute the Fibonacci sequence, I can get a huge speedup from parallelisation. Likewise, if I do element-wise multiplication in a way that takes no advantage of vectorization, I get a similar speedup for the parallel version:

import numpy as np
import os
from timeit import timeit
from multiprocessing import Pool

def fib(dummy):
    n = [1,1]
    for ii in xrange(100000):
        n.append(n[-1]+n[-2])

def silly_mult(matrix):
    for row in matrix:
        for val in row:
            val * val

if __name__ == '__main__':

    dt = timeit(lambda: map(fib, xrange(10)), number=10)
    print "Fibonacci, non-parallel: %.3f" %dt

    matrices = [np.random.randn(1000,1000) for ii in xrange(10)]
    dt = timeit(lambda: map(silly_mult, matrices), number=10)
    print "Silly matrix multiplication, non-parallel: %.3f" %dt

    # after importing numpy, reset the CPU affinity of the parent process so
    # that it will use all CPUS
    os.system("taskset -p 0xff %d" % os.getpid())

    pool = Pool(8)

    dt = timeit(lambda: pool.map(fib,xrange(10)), number=10)
    print "Fibonacci, parallel: %.3f" %dt

    dt = timeit(lambda: pool.map(silly_mult, matrices), number=10)
    print "Silly matrix multiplication, parallel: %.3f" %dt

Output:

$ python tmp.py
Fibonacci, non-parallel: 32.449
Silly matrix multiplication, non-parallel: 40.084
pid 29528's current affinity mask: 1
pid 29528's new affinity mask: ff
Fibonacci, parallel: 9.462
Silly matrix multiplication, parallel: 12.163

More Related Contents:

  1. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  2. Can’t pickle when using multiprocessing Pool.map()
  3. Use numpy array in shared memory for multiprocessing
  4. Shared-memory objects in multiprocessing
  5. Why does multiprocessing use only a single core after I import numpy?
  6. Is shared readonly data copied to different processes for multiprocessing?
  7. Keyboard Interrupts with python’s multiprocessing Pool
  8. Python Process Pool non-daemonic?
  9. Combine Pool.map with shared memory Array in Python multiprocessing
  10. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  11. Filling a queue and managing multiprocessing in python
  12. Can I use a multiprocessing Queue in a function called by Pool.imap?
  13. Parallelizing a Numpy vector operation
  14. multiprocessing.Pool() slower than just using ordinary functions
  15. How do you pass a Queue reference to a function managed by pool.map_async()?
  16. Passing multiple parameters to pool.map() function in Python [duplicate]
  17. How to share numpy random state of a parent process with child processes?
  18. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  19. How to let Pool.map take a lambda function
  20. Python: Writing to a single file with queue while using multiprocessing Pool
  21. Python multiprocessing – tracking the process of pool.map operation
  22. Selecting multiple slices from a numpy array at once
  23. Find the most frequent number in a NumPy array
  24. Getting the indices of several elements in a NumPy array at once
  25. Using Python’s Multiprocessing module to execute simultaneous and separate SEAWAT/MODFLOW model runs
  26. Multiprocessing – Pipe vs Queue
  27. OSError 38 [Errno 38] with multiprocessing
  28. Better way to shuffle two numpy arrays in unison
  29. numpy – evaluate function on a grid of points
  30. How to convert a 3d numpy array to 2d

Leave a Comment