In addition to Martijn’s explanation: A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it’s not nested within another function, and it is a “real” function with a name, not a lambda function. So, to pickle your defaultdict, create it with … Read more
Yes, the fact that it’s a class member is a problem: >>> class Foo(): … Bar = namedtuple(‘Bar’, [‘x’,’y’]) … def baz(self): … b = Foo.Bar(x=2, y=3) … print(type(b)) … >>> a = Foo() >>> a.baz() <class ‘__main__.Bar’> The problem is that when namedtuple() returns a type object, it isn’t aware of the fact that … Read more
I know this is a very old question, but I have never explicitly seen a satisfactory solution to this question other than the obvious, and most likely correct, answer to re-structure your code. Unfortunately, it is not always practical to do such a thing, in which case as a very last resort, it is possible … Read more
From the docs: Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RuntimeError will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit(). Although your trie implementation may be simple, it uses recursion and can lead to issues when converting to a persistent data structure. … Read more
I prefer JSON over pickle for my serialization. Unpickling can run arbitrary code, and using pickle to transfer data between programs or store data between sessions is a security hole. JSON does not introduce a security hole and is standardized, so the data can be accessed by programs in different languages if you ever need … Read more
Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script. As for where the pickled information is stored, usually one would do: with open(‘filename’, ‘wb’) as f: var = … Read more
multiprocessing makes some bad choices about pickling. Don’t get me wrong, it makes some good choices that enable it to pickle certain types so they can be used in a pool’s map function. However, since we have dill that can do the pickling, multiprocessing’s own pickling becomes a bit limiting. Actually, if multiprocessing were to … Read more
The issue is that you’re pickling objects defined in Settings by actually running the ‘Settings’ module, then you’re trying to unpickle the objects from the GUI module. Remember that pickle doesn’t actually store information about how a class/object is constructed, and needs access to the class when unpickling. See wiki on using Pickle for more … Read more
You’ll have to tell pickle.load() how to convert Python bytestring data to Python 3 strings, or you can tell pickle to leave them as bytes. The default is to try and decode all string data as ASCII, and that decoding fails. See the pickle.load() documentation: Optional keyword arguments are fix_imports, encoding and errors, which are … Read more
I’ve compared performance (space and time) for a number of ways to store numpy arrays. Few of them support multiple arrays per file, but perhaps it’s useful anyway. Npy and binary files are both really fast and small for dense data. If the data is sparse or very structured, you might want to use npz … Read more