multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’

by

The http.client.HTTPResponse-object you get back from urlopen() has a _io.BufferedReader-object attached, and this object cannot be pickled. 

pickle.dumps(urllib.request.urlopen('http://www.python.org').fp)
Traceback (most recent call last):
...
    pickle.dumps(urllib.request.urlopen('http://www.python.org').fp)
TypeError: cannot serialize '_io.BufferedReader' object

multiprocessing.Pool will need to pickle (serialize) the results to send it back to the parent process and this fails here. Since dummy uses threads instead of processes, there will be no pickling, because threads in the same process share their memory naturally.

A general solution to this TypeError is:

  1. read out the buffer and save the content (if needed)
  2. remove the reference to '_io.BufferedReader' from the object you are trying to pickle

In your case, calling .read() on the http.client.HTTPResponse will empty and remove the buffer, so a function for converting the response into something pickleable could simply do this:

def read_buffer(response):
    response.text = response.read()
    return response

Example:

r = urllib.request.urlopen('http://www.python.org')
r = read_buffer(r)
pickle.dumps(r)
# Out: b'\x80\x03chttp.client\nHTTPResponse\...

Before you consider this approach, make sure you really want to use multiprocessing instead of multithreading. For I/O-bound tasks like you have it here, multithreading would be sufficient, since most of the time is spend in waiting (no need for cpu-time) for the response anyway. Multiprocessing and the IPC involved also introduces substantial overhead.

More Related Contents:

  1. How to use multiprocessing pool.map with multiple arguments
  2. Python multiprocessing PicklingError: Can’t pickle
  3. Sharing a result queue among several processes
  4. multiprocessing: Understanding logic behind `chunksize`
  5. Shared variable in python’s multiprocessing
  6. multiprocessing.Pool: What’s the difference between map_async and imap?
  7. Script using multiprocessing module does not terminate
  8. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  9. Python multiprocessing.Pool: AttributeError
  10. How to get the return value of a function passed to multiprocessing.Process?
  11. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  12. how to to terminate process using python’s multiprocessing
  13. multiprocessing pool example does not work and freeze the kernel
  14. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  15. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  16. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  17. Python multiprocessing application is getting stuck in docker container
  18. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  19. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  20. How can I abort a task in a multiprocessing.Pool after a timeout?
  21. How can I share a class between processes?
  22. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  23. Downloading a picture via urllib and python
  24. Python: execute cat subprocess in parallel
  25. AttributeError: ‘module’ object has no attribute ‘urlopen’
  26. Multiprocessing working in Python but not in iPython
  27. Python: Get HTTP headers from urllib2.urlopen call?
  28. Python multiprocessing – tracking the process of pool.map operation
  29. Can I get a return value from multiprocessing.Process?
  30. Tensorflow and Multiprocessing: Passing Sessions

Leave a Comment