How to estimate how much memory a Pandas’ DataFrame will need?

by

df.memory_usage() will return how many bytes each column occupies:

>>> df.memory_usage()

Row_ID            20906600
Household_ID      20906600
Vehicle           20906600
Calendar_Year     20906600
Model_Year        20906600
...

To include indexes, pass index=True.

So to get overall memory consumption:

>>> df.memory_usage(index=True).sum()
731731000

Also, passing deep=True will enable a more accurate memory usage report, that accounts for the full usage of the contained objects.

This is because memory usage does not include memory consumed by elements that are not components of the array if deep=False (default case).

More Related Contents:

  1. Split / Explode a column of dictionaries into separate columns with pandas
  2. How to split a dataframe string column into two columns?
  3. Comparing two dataframes and getting the differences [duplicate]
  4. plot different color for different categorical levels using matplotlib
  5. Pandas: Get Dummies
  6. Get the name of a pandas DataFrame
  7. Calculate weighted average using a pandas/dataframe
  8. Comparing previous row values in Pandas DataFrame
  9. What is the difference between join and merge in Pandas?
  10. Ambiguity in Pandas Dataframe / Numpy Array “axis” definition
  11. Pandas Dataframe: Replacing NaN with row average
  12. How to read a .xlsx file using the pandas Library in iPython?
  13. Counting unique values in a column in pandas dataframe like in Qlik?
  14. ValueError: Length of values does not match length of index | Pandas DataFrame.unique()
  15. Pandas Groupby / List to Multiple Rows
  16. How to reorder indexed rows based on a list in Pandas data frame
  17. How can I count the number of consecutive TRUEs in a DataFrame?
  18. Why apply sometimes isn’t faster than for-loop in a Pandas dataframe?
  19. Convert month int to month name in Pandas
  20. How to plot percentage with seaborn distplot / histplot / displot
  21. Pandas Plotting Display all date values on x-axis (matplolib only displays few values) formatted as MMM-YYYY [duplicate]
  22. Week of a month pandas
  23. MemoryError when I merge two Pandas data frames
  24. Add a sequence number to each element in a group using python
  25. How to use sklearn fit_transform with pandas and return dataframe instead of numpy array?
  26. pandas convert from datetime to integer timestamp
  27. geopandas point in polygon
  28. mean from pandas and numpy differ
  29. Rename columns in Pandas based on a dictionary
  30. When is it appropriate to use df.value_counts() vs df.groupby(‘…’).count()?

Leave a Comment