Share list between process in python server

by

The problem is that you’re letting the main process finish its execution immediately after you start the worker process. When the process that created the multiprocessing.Manager finishes its execution, the Manager server gets shut down, which means your shared list object is now useless. This happens because the Manager object registers it’s shutdown function as a “finalizer” with the multiprocessing module, which means it will be run just before the process exits. Here’s the code that registers it, in BaseManager.__init__:

    # register a finalizer
    self._state.value = State.STARTED
    self.shutdown = util.Finalize(
        self, type(self)._finalize_manager,
        args=(self._process, self._address, self._authkey,
              self._state, self._Client),
        exitpriority=0
        )

Here’s the code that actually does the shut down:

@staticmethod
def _finalize_manager(process, address, authkey, state, _Client):
    '''
    Shutdown the manager process; will be registered as a finalizer
    '''
    if process.is_alive():
        util.info('sending shutdown message to manager')
        try:
            conn = _Client(address, authkey=authkey)
            try:
                dispatch(conn, None, 'shutdown')
            finally:
                conn.close()
        except Exception:
            pass

        process.join(timeout=1.0)
        if process.is_alive():
            util.info('manager still alive')
            if hasattr(process, 'terminate'):
                util.info('trying to `terminate()` manager process')
                process.terminate()
                process.join(timeout=0.1)
                if process.is_alive():
                    util.info('manager still alive after terminate')

    state.value = State.SHUTDOWN
    try:
        del BaseProxy._address_to_local[address]
    except KeyError:
        pass

The fix is simple – don’t let the main process complete immediately you start the process that runs the UDP server, by calling server_process.join():

import multiprocessing
from socketserver import UDPServer, ForkingMixIn, DatagramRequestHandler
from socket import socket, AF_INET, SOCK_DGRAM
from settings import host, port, number_of_connections

class ChatHandler(DatagramRequestHandler):

    def handle(self):
        cur_process = multiprocessing.current_process()
        data = self.request[0].strip()
        socket = self.request[1]
        ChatHandler.clients.append(self.client_address) # error here
        print(ChatHandler.clients)


class ChatServer(ForkingMixIn, UDPServer):
    pass


if __name__ == '__main__':
    server = ChatServer((host, port), ChatHandler)
    ChatHandler.clients = multiprocessing.Manager().list()
    server_process = multiprocessing.Process(target=server.serve_forever)
    server_process.daemon = False
    server_process.start()
    server_process.join() # This fixes the issue.

More Related Contents:

  1. How to run functions in parallel?
  2. multiprocessing.Pool: When to use apply, apply_async or map?
  3. Shared-memory objects in multiprocessing
  4. Shared memory in multiprocessing
  5. Dead simple example of using Multiprocessing Queue, Pool and Locking
  6. Shared variable in python’s multiprocessing
  7. multiprocessing.Pool: What’s the difference between map_async and imap?
  8. Python Multiprocessing: Handling Child Errors in Parent
  9. Processing single file from multiple processes
  10. How to use a multiprocessing.Manager()?
  11. Solving embarassingly parallel problems using Python multiprocessing
  12. Log output of multiprocessing.Process
  13. multiprocessing vs multithreading vs asyncio in Python 3
  14. pandas multiprocessing apply
  15. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  16. Multiprocessing : use tqdm to display a progress bar
  17. Appending to the same list from different processes using multiprocessing
  18. Exception thrown in multiprocessing Pool not detected
  19. Using the multiprocessing module for cluster computing
  20. multiprocessing.Pool() slower than just using ordinary functions
  21. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  22. Chrome crashes after several hours while multiprocessing using Selenium through Python
  23. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  24. What is the difference between multiprocessing and subprocess?
  25. Python Multiprocessing Lib Error (AttributeError: __exit__)
  26. Show the progress of a Python multiprocessing pool imap_unordered call?
  27. How to process requests from multiiple users using ML model and FastAPI?
  28. Easiest way to read csv files with multiprocessing in Pandas
  29. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  30. Using python multiprocessing with different random seed for each process

Leave a Comment