Why can’t generators be pickled?

by

There is lots of information about this available. For the “official word” on the issue, read the (closed) Python bugtracker issue.

The core reasoning, by one of the people who made the decision, is detailed on this blog:

Since a generator is essentially a souped-up function, we would need to save its bytecode, which is not guarantee to be backward-compatible between Python’s versions, and its frame, which holds the state of the generator such as local variables, closures and the instruction pointer. And this latter is rather cumbersome to accomplish, since it basically requires to make the whole interpreter picklable. So, any support for pickling generators would require a large number of changes to CPython’s core.

Now if an object unsupported by pickle (e.g., a file handle, a socket, a database connection, etc) occurs in the local variables of a generator, then that generator could not be pickled automatically, regardless of any pickle support for generators we might implement. So in that case, you would still need to provide custom __getstate__ and __setstate__ methods. This problem renders any pickling support for generators rather limited.

And two suggested workarounds are mentioned:

Anyway, if you need for a such feature, then look into Stackless Python which does all the above. And since Stackless’s interpreter is picklable, you also get process migration for free. This means you can interrupt a tasklet (the name for Stackless’s green threads), pickle it, send the pickle to a another machine, unpickle it, resume the tasklet, and voilà you’ve just migrated a process. This is freaking cool feature!

But in my humble opinion, the best solution to this problem to the rewrite the generators as simple iterators (i.e., one with a __next__ method). Iterators are easy and efficient space-wise to pickle because their state is explicit. You would still need to handle objects representing some external state explicitly however; you cannot get around this.

More Related Contents:

  1. Understanding generators in Python
  2. Lazy Method for Reading Big File in Python?
  3. Difference between Python’s Generators and Iterators
  4. How do I know if a generator is empty from the start?
  5. Resetting generator object in Python
  6. Serializing class instance to JSON
  7. Using pickle.dump – TypeError: must be str, not bytes
  8. best way to preserve numpy arrays on disk
  9. Can iterators be reset in Python?
  10. Python serialization – Why pickle?
  11. Converting “yield from” statement to Python 2.7 code
  12. How to len(generator()) [duplicate]
  13. Python generator that groups another iterable into groups of N
  14. Python: Can’t pickle type X, attribute lookup failed
  15. Length of generator output [duplicate]
  16. Get the nth item of a generator in Python
  17. How to get the n next values of a generator in a list (python)
  18. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  19. How can I generate all possible IPs from a list of ip ranges in Python?
  20. Best way to receive the ‘return’ value from a python generator
  21. Python: using a recursive algorithm as a generator
  22. Python Multiprocessing Lib Error (AttributeError: __exit__)
  23. Return in generator together with yield
  24. What does the “yield” keyword do in Python?
  25. Why do I get “Pickle – EOFError: Ran out of input” reading an empty file?
  26. How to clone a Python generator object?
  27. Why do list comprehensions write to the loop variable, but generators don’t? [duplicate]
  28. Python: powerset of a given set with generators [duplicate]
  29. ModuleNotFoundError when trying to unpickle a model in Flask app
  30. Python using exceptions for control flow considered bad?

Leave a Comment