I got multiprocessing to work from within a Jupyter notebook on Windows by saving my function in a separate .py file and including that file in my notebook. Example: f.py: def f(name, output): output.put(‘hello {0}’.format(name)) return Code in Jupyter notebook: from multiprocessing import Process, Queue #Having the function definition here results in #AttributeError: Can’t get … Read more
Most of the time it’s not really desirable to pass the reference of an existing object to another process. Instead you create your class you want to share between processes: class MySharedClass: # stuff… Then you make a proxy manager like this: import multiprocessing.managers as m class MyManager(m.BaseManager): pass # Pass is really enough. Nothing … Read more
I tried something, and it seems to work. I put preload_app = True in my gunicorn.conf and now the lock seems to be shared. I am still looking into exactly what’s happening here but for now this is good enough, YMMV.
Just thought I would add an actual answer to make it clear for others. Quoting the answer from aix in this question: What happens is that on Unix every worker process inherits the same state of the random number generator from the parent process. This is why they generate identical pseudo-random sequences. Use the random.seed() … Read more
No process can stop another short of brute force os.kill()-like sledgehammers. Don’t go there. To do this sanely, you need to rework your basic approach: the main process and the worker processes need to communicate with each other. I’d flesh it out, but the example so far is too bare-bones to make it useful. For … Read more
The solution you suggest is a good one: create your processes manually such that you have explicit access to their stdout/stderr file handles. You can then create a socket to communicate with the sub-process and use multiprocessing.connection over that socket (multiprocessing.Pipe creates the same type of connection object, so this should give you all the … Read more
Windows lacks a fork() system call, which duplicates current process. This has many implications, including those listed on the windows multiprocessing documentation page. More specifically: Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as … Read more
If I have to guess what’s primarily wrong with your code, I’d say it’s in passing your input_rows to your process function insert() – the way multiprocessing.Pool.apply_async() works is to unpack the arguments passed to it, so your insert() function actually retreives 100 arguments instead of one argument with a list of 100 elements. This … Read more
Ah here comes yet another discussion about the GIL. Well here’s the thing. Fetching content with urllib2 is going to be mostly IO-bound. Native threading AND multiprocessing will both have the same performance when the task is IO-bound (threading only becomes a problem when it’s CPU-bound). Yes you can speed it up, I’ve done it … Read more
Linux uses copy-on-write when subprocesses are os.forked. To demonstrate: import multiprocessing as mp import numpy as np import logging import os logger = mp.log_to_stderr(logging.WARNING) def free_memory(): total = 0 with open(‘/proc/meminfo’, ‘r’) as f: for line in f: line = line.strip() if any(line.startswith(field) for field in (‘MemFree’, ‘Buffers’, ‘Cached’)): field, amount, unit = line.split() amount … Read more