Your usage of the shared dictionary is identical to and can be re-written as this: temp = shared_object[‘nested’] temp.update({‘second_key’: ‘second_value’}) When you request the value of the nested key from the shared dictionary, it is stored in the current process’s memory space. This means that any changes you do to this dictionary will not be … Read more
I think this is a regression in Python 3.7.2 as described here. It seems to only affect users when running in a virtualenv. For the time being you can work-around it by doing what’s described in this comment on the bug thread. import _winapi import multiprocessing.spawn multiprocessing.spawn.set_executable(_winapi.GetModuleFileName(0)) That will force the subprocesses to spawn using … Read more
Note that I’m using pathos.multiprocessing instead of multiprocessing. It’s just a fork of multiprocessing that enables you to do map functions with multiple inputs, has much better serialization, and allows you to execute map calls anywhere (not just in __main__). You could use multiprocessing to do the below as well, however the code would be … Read more
Twisted has its own event-driven way of running subprocesses which is (in my humble, but correct, opinion) better than the multiprocessing module. The core API is spawnProcess, but tools like ampoule provide higher-level wrappers over it. If you use spawnProcess, you will be able to handle output from subprocesses in the same way you’d handle … Read more
Using Pool: import os import pandas as pd from multiprocessing import Pool # wrap your csv importer in a function that can be mapped def read_csv(filename): ‘converts a filename to a pandas dataframe’ return pd.read_csv(filename) def main(): # get a list of file names files = os.listdir(‘.’) file_list = [filename for filename in files if … Read more
Multiprocessing pools implement a queue for you. Just use a pool method that returns the worker return value to the caller. imap works well: import multiprocessing import re def mp_worker(filename): with open(filename) as f: text = f.read() m = re.findall(“x+”, text) count = len(max(m, key=len)) return filename, count def mp_handler(): p = multiprocessing.Pool(32) with open(‘infilenamess.txt’) … Read more
You are looking to do some embarrassingly parallel work using multiple processes, so why not use a Pool? A Pool will take care of starting up the processes, retrieving the results, and returning the results to you. I use pathos, which has a fork of multiprocessing, because it has much better serialization than the version … Read more
The problem is not with the code / Python 3.6, it is with Spyder. After some investigation I found that the code runs fine when executed in an external system terminal but not when run in Spyder’s IPython console. I was able to dump the contents of spec and assign them to a variable that … Read more
To use 100% of all cores, do not create and destroy new processes. Create a few processes per core and link them with a pipeline. At the OS-level, all pipelined processes run concurrently. The less you write (and the more you delegate to the OS) the more likely you are to use as many resources … Read more
First, you should rather not load your model every time a request arrives, but rahter have it loaded once at startup (you could use the startup event for this) and store it on the app instance—using the generic app.state attribute (see implementation of State too)—which you can later retrieve, as described here and here. For … Read more