Yes, python can pickle lambda functions… but only if you have something that uses copy_reg to register how to pickle lambda functions — the package dill loads the copy_reg you need into the pickle registry for you, when you import dill. Python 2.7.8 (default, Jul 13 2014, 02:29:54) [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] … Read more
The output file needs to be opened in binary mode: f = open(‘varstor.txt’,’w’) needs to be: f = open(‘varstor.txt’,’wb’)
Pickle save: try: import cPickle as pickle except ImportError: # Python 3.x import pickle with open(‘data.p’, ‘wb’) as fp: pickle.dump(data, fp, protocol=pickle.HIGHEST_PROTOCOL) See the pickle module documentation for additional information regarding the protocol argument. Pickle load: with open(‘data.p’, ‘rb’) as fp: data = pickle.load(fp) JSON save: import json with open(‘data.json’, ‘w’) as fp: json.dump(data, fp) … Read more
I could not use the codes posted so far because the codes using “multiprocessing.Pool” do not work with lambda expressions and the codes not using “multiprocessing.Pool” spawn as many processes as there are work items. I adapted the code s.t. it spawns a predefined amount of workers and only iterates through the input list if … Read more
You could serialise the function bytecode and then reconstruct it on the caller. The marshal module can be used to serialise code objects, which can then be reassembled into a function. ie: import marshal def foo(x): return x*x code_string = marshal.dumps(foo.__code__) Then in the remote process (after transferring code_string): import marshal, types code = marshal.loads(code_string) … Read more
The basic problem is that the JSON encoder json.dumps() only knows how to serialize a limited set of object types by default, all built-in types. List here: https://docs.python.org/3.3/library/json.html#encoders-and-decoders One good solution would be to make your class inherit from JSONEncoder and then implement the JSONEncoder.default() function, and make that function emit the correct JSON for … Read more
Try this: import pickle a = {‘hello’: ‘world’} with open(‘filename.pickle’, ‘wb’) as handle: pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL) with open(‘filename.pickle’, ‘rb’) as handle: b = pickle.load(handle) print(a == b) There’s nothing about the above solution that is specific to a dict object. This same approach will will work for many Python objects, including instances of arbitrary classes … Read more
Two additions to Tim Peters’ accepted answer. First, you need not store the number of items you pickled separately if you stop loading when you hit the end of the file: def loadall(filename): with open(filename, “rb”) as f: while True: try: yield pickle.load(f) except EOFError: break items = loadall(myfilename) This assumes the file contains only … Read more
The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it “easy” or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method. For example, Steven Bethard’s … Read more
Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module. This piece of code: import multiprocessing as mp class Foo(): @staticmethod def work(self): pass if __name__ == ‘__main__’: pool = mp.Pool() foo = Foo() pool.apply_async(foo.work) pool.close() pool.join() yields an error … Read more