How to reversibly store and load a Pandas dataframe to/from disk

by

The easiest way is to pickle it using to_pickle:

df.to_pickle(file_name)  # where to save it, usually as a .pkl

Then you can load it back using:

df = pd.read_pickle(file_name)

Note: before 0.11.1 save and load were the only way to do this (they are now deprecated in favor of to_pickle and read_pickle respectively).

Another popular choice is to use HDF5 (pytables) which offers very fast access times for large datasets:

import pandas as pd
store = pd.HDFStore('store.h5')

store['df'] = df  # save it
store['df']  # load it

More advanced strategies are discussed in the cookbook.

Since 0.13 there’s also msgpack which may be be better for interoperability, as a faster alternative to JSON, or if you have python object/text-heavy data (see this question).

More Related Contents:

  1. error when applying function to a series object
  2. How to deal with SettingWithCopyWarning in Pandas
  3. Groupby value counts on the dataframe pandas
  4. Using Pandas to pd.read_excel() for multiple worksheets of the same workbook
  5. How to add pandas data to an existing csv file?
  6. How to explode a list inside a Dataframe cell into separate rows
  7. How to print pandas DataFrame without index
  8. Reversing ‘one-hot’ encoding in Pandas
  9. How to loop over grouped Pandas dataframe?
  10. Update a dataframe in pandas while iterating row by row
  11. replace column values in one dataframe by values of another dataframe
  12. Conditionally format Python pandas cell
  13. Pandas expand rows from list data available in column
  14. How to concatenate multiple column values into a single column in Pandas dataframe
  15. How can I use cumsum within a group in Pandas?
  16. Pandas dataframe – running sum with reset
  17. Why does df.apply(tuple) work but not df.apply(list)?
  18. Merge a list of pandas dataframes
  19. Checking if particular value (in cell) is NaN in pandas DataFrame not working using ix or iloc
  20. Updating a value in a pandas dataframe in an iterrows loop
  21. Move column by name to front of table in pandas
  22. Convert List to Pandas Dataframe Column
  23. How to repeat a Pandas DataFrame?
  24. Get column name where value is something in pandas dataframe
  25. Is there a way to speed up handling large CSVs and dataframes in python?
  26. Remove ends of string entries in pandas DataFrame column
  27. Collapsing rows in a Pandas dataframe if all rows have only one value in their columns
  28. start index at 1 for Pandas DataFrame
  29. Multi Index Sorting in Pandas
  30. Pythonic/efficient way to strip whitespace from every Pandas Data frame cell that has a stringlike object in it

Leave a Comment