multiprocessing in python – sharing large object (e.g. pandas dataframe) between multiple processes

by

The first argument to Value is typecode_or_type. That is defined as:

typecode_or_type determines the type of the returned object: it is
either a ctypes type or a one character typecode of the kind used by
the array module. *args is passed on to the constructor for the type.

Emphasis mine. So, you simply cannot put a pandas dataframe in a Value, it has to be a ctypes type.

You could instead use a multiprocessing.Manager to serve your singleton dataframe instance to all of your processes. There’s a few different ways to end up in the same place – probably the easiest is to just plop your dataframe into the manager’s Namespace.

from multiprocessing import Manager

mgr = Manager()
ns = mgr.Namespace()
ns.df = my_dataframe

# now just give your processes access to ns, i.e. most simply
# p = Process(target=worker, args=(ns, work_unit))

Now your dataframe instance is accessible to any process that gets passed a reference to the Manager. Or just pass a reference to the Namespace, it’s cleaner.

One thing I didn’t/won’t cover is events and signaling – if your processes need to wait for others to finish executing, you’ll need to add that in. Here is a page with some Event examples which also cover with a bit more detail how to use the manager’s Namespace.

(note that none of this addresses whether multiprocessing is going to result in tangible performance benefits, this is just giving you the tools to explore that question)

More Related Contents:

  1. pandas multiprocessing apply
  2. Easiest way to read csv files with multiprocessing in Pandas
  3. How to iterate over rows in a DataFrame in Pandas
  4. Selecting with complex criteria from pandas.DataFrame
  5. Stacked Bar Chart with Centered Labels
  6. Combine two pandas Data Frames (join on a common column)
  7. pandas .at versus .loc
  8. Can pandas groupby aggregate into a list, rather than sum, mean, etc?
  9. timedelta to string type in pandas dataframe
  10. Can I use a multiprocessing Queue in a function called by Pool.imap?
  11. drop_duplicates not working in pandas?
  12. Adding values for missing data combinations in Pandas
  13. python pandas – creating a column which keeps a running count of consecutive values
  14. Turn Pandas Multi-Index into column
  15. Why is pandas.to_datetime slow for non standard time format such as ‘2014/12/31’
  16. What is the fastest and most efficient way to append rows to a DataFrame?
  17. Converting pandas column of comma-separated strings into dummy variables
  18. How to get the return value of a function passed to multiprocessing.Process?
  19. How to plot different groups of data from a dataframe into a single figure
  20. How to share numpy random state of a parent process with child processes?
  21. how to concat two data frames with different column names in pandas? – python
  22. Check if rows in one dataframe exist in another dataframe
  23. Pandas – Interleave / Zip two DataFrames by row
  24. How indexing works in Pandas?
  25. Pandas custom function to find whether it is the 1st, 2nd etc Monday, Tuesday, etc – all suggestions welcome
  26. Normalize DataFrame by group
  27. Color a scatter plot by Column Values
  28. pandas extract year from datetime: df[‘year’] = df[‘date’].year is not working
  29. pandas.read_csv FileNotFoundError: File b’\xe2\x80\xaa’ despite correct path
  30. Pandas: Selecting rows based on value counts of a particular column

Leave a Comment