How to pickle a python function with its dependencies?

by

Updated Sep 2020: See the comment by @ogrisel below. The developers of PiCloud moved to Dropbox shortly after I wrote the original version of this answer in 2013, though a lot of folks are still using the cloudpickle module seven years later. The module made its way to Apache Spark, where it has continued to be maintained and improved. I’m updating the example and background text below accordingly.

Cloudpickle

The cloudpickle package is able to pickle a function, method, class, or even a lambda, as well as any dependencies. To try it out, just pip install cloudpickle and then:

import cloudpickle

def foo(x):
    return x*3

def bar(z):
    return foo(z)+1

x = cloudpickle.dumps(bar)
del foo
del bar

import pickle

f = pickle.loads(x)
print(f(3))  # displays "10"

In other words, just call cloudpickle.dump() or cloudpickle.dumps() the same way you’d use pickle.*, then later use the native pickle.load() or pickle.loads() to thaw.

Background

PiCcloud.com released the cloud python package under the LGPL, and other open-source projects quickly started using it (google for cloudpickle.py to see a few). The folks at picloud.com had an incentive to put the effort into making general-purpose code pickling work — their whole business was built around it. The idea was that if you had cpu_intensive_function() and wanted to run it on Amazon’s EC2 grid, you just replaced:

cpu_intensive_function(some, args)

with:

cloud.call(cpu_intensive_function, some, args)

The latter used cloudpickle to pickle up any dependent code and data, shipped it to EC2, ran it, and returned the results to you when you called cloud.result().

Picloud billed in millisecond increments, it was cheap as heck, and I used it all the time for Monte Carlo simulations and financial time series analysis, when I needed hundreds of CPU cores for just a few seconds each. Years later, I still can’t say enough good things about it and I didn’t even work there.

More Related Contents:

  1. Is there an easy way to pickle a python function (or otherwise serialize its code)?
  2. Trying to use a returned value without the other parts of a function [closed]
  3. How to use variables from a function in a different function
  4. What causes my function to return None at the end?
  5. Passing a dictionary to a function as keyword parameters
  6. Why does using `arg=None` fix Python’s mutable default argument issue?
  7. unpacking function argument [duplicate]
  8. What can multiprocessing and dill do together?
  9. What is the naming convention in Python for variable and function?
  10. Difference between methods and functions, in Python compared to C++
  11. How do I call a function from another .py file?
  12. Python function as a function argument?
  13. Overloaded functions in Python
  14. Pickle or json?
  15. Hitting Maximum Recursion Depth Using Pickle / cPickle
  16. Setting variables with exec inside a function
  17. How can I make a deepcopy of a function in Python?
  18. Function chaining in Python
  19. PicklingError: Can’t pickle : it’s not the same object as decimal.Decimal
  20. Pickle incompatibility of numpy arrays between Python 2 and 3
  21. Efficient evaluation of a function at every cell of a NumPy array
  22. In Python, what does ” mean?
  23. break and continue in function
  24. Python 3 – Can pickle handle byte objects larger than 4GB?
  25. Python functions with multiple parameter brackets
  26. function is not defined error in Python
  27. Why is “None” printed after my function’s output?
  28. Is it possible to not return anything from a function in python? (What happens if the function reaches the end without `return`ing?)
  29. Python: why pickle?
  30. Why can’t generators be pickled?

Leave a Comment