Python pickling after changing a module’s directory

by

As pickle’s docs say, in order to save and restore a class instance (actually a function, too), you must respect certain constraints:

pickle can save and restore class
instances transparently, however the
class definition must be importable
and live in the same module as when
the object was stored

whyteboard.tools is not the “the same module as” tools (even though it can be imported by import tools by other modules in the same package, it ends up in sys.modules as sys.modules['whyteboard.tools']: this is absolutely crucial, otherwise the same module imported by one in the same package vs one in another package would end up with multiple and possibly conflicting entries!).

If your pickle files are in a good/advanced format (as opposed to the old ascii format that’s the default only for compatibility reasons), migrating them once you perform such changes may in fact not be quite as trivial as “editing the file” (which is binary &c…!), despite what another answer suggests. I suggest that, instead, you make a little “pickle-migrating script”: let it patch sys.modules like this…:

import sys
from whyteboard import tools

sys.modules['tools'] = tools

and then cPickle.load each file, del sys.modules['tools'], and cPickle.dump each loaded object back to file: that temporary extra entry in sys.modules should let the pickles load successfully, then dumping them again should be using the right module-name for the instances’ classes (removing that extra entry should make sure of that).

More Related Contents:

  1. Saving an Object (Data persistence)
  2. Saving and loading multiple objects in pickle file?
  3. How can I use pickle to save a dict (or any other Python object)?
  4. Multiprocessing: How to use Pool.map on a function defined in a class?
  5. Storing Python dictionaries
  6. Can Python pickle lambda functions?
  7. Unpickling a python 2 object with python 3
  8. Unable to load files using pickle and multiple modules
  9. How can I pickle a dynamically created nested class in python?
  10. Can’t pickle defaultdict
  11. How to read pickle file?
  12. Preserve custom attributes when pickling subclass of numpy array
  13. What causes the error “_pickle.UnpicklingError: invalid load key, ‘ ‘.”?
  14. What is faster – Loading a pickled dictionary object or Loading a JSON file – to a dictionary? [closed]
  15. multiprocessing.Pool – PicklingError: Can’t pickle : attribute lookup thread.lock failed
  16. Is pickle file of python cross-platform?
  17. Saving and loading objects and using pickle
  18. ImportError: No module named ‘pandas.indexes’
  19. What is the difference between rb and r+b modes in file objects [duplicate]
  20. pickle – putting more than 1 object in a file? [duplicate]
  21. pickle.load() raising EOFError in Windows
  22. Python Pickling Dictionary EOFError
  23. Common use-cases for pickle in Python
  24. Dump a list in a pickle file and retrieve it back later [closed]
  25. Python 3.7 Error: Unsupported Pickle Protocol 5
  26. Python pickle protocol choice?
  27. Understanding Pickling in Python
  28. Pickle with custom classes
  29. How to hash a large object (dataset) in Python?
  30. cProfile causes pickling error when running multiprocessing Python code

Leave a Comment