When you use multiprocessing.dummy
, you’re using threads, not processes:
multiprocessing.dummy
replicates the API ofmultiprocessing
but is no
more than a wrapper around thethreading
module.
That means you’re restricted by the Global Interpreter Lock (GIL), and only one thread can actually execute CPU-bound operations at a time. That’s going to keep you from fully utilizing your CPUs. If you want get full parallelism across all available cores, you’re going to need to address the pickling issue you’re hitting with multiprocessing.Pool
.
Note that multiprocessing.dummy
might still be useful if the work you need to parallelize is IO bound, or utilizes a C-extension that releases the GIL. For pure Python code, however, you’ll need multiprocessing
.