Data size in memory vs. on disk

by

Python Object Data Size

If the data is stored in some python object, there will be a little more data attached to the actual data in memory.

This may be easily tested.

The size of data in various forms

It is interesting to note how, at first, the overhead of the python object is significant for small data, but quickly becomes negligible.

Here is the iPython code used to generate the plot

%matplotlib inline
import random
import sys
import array
import matplotlib.pyplot as plt

max_doubles = 10000

raw_size = []
array_size = []
string_size = []
list_size = []
set_size = []
tuple_size = []
size_range = range(max_doubles)

# test double size
for n in size_range:
    double_array = array.array('d', [random.random() for _ in xrange(n)])
    double_string = double_array.tostring()
    double_list = double_array.tolist()
    double_set = set(double_list)
    double_tuple = tuple(double_list)

    raw_size.append(double_array.buffer_info()[1] * double_array.itemsize)
    array_size.append(sys.getsizeof(double_array))
    string_size.append(sys.getsizeof(double_string))
    list_size.append(sys.getsizeof(double_list))
    set_size.append(sys.getsizeof(double_set))
    tuple_size.append(sys.getsizeof(double_tuple))

# display
plt.figure(figsize=(10,8))
plt.title('The size of data in various forms', fontsize=20)
plt.xlabel('Data Size (double, 8 bytes)', fontsize=15)
plt.ylabel('Memory Size (bytes)', fontsize=15)
plt.loglog(
    size_range, raw_size, 
    size_range, array_size, 
    size_range, string_size,
    size_range, list_size,
    size_range, set_size,
    size_range, tuple_size
)
plt.legend(['Raw (Disk)', 'Array', 'String', 'List', 'Set', 'Tuple'], fontsize=15, loc="best")

More Related Contents:

  1. How can I explicitly free memory in Python?
  2. How to get current CPU and RAM usage in Python?
  3. Is it possible to dereference variable id’s?
  4. when does Python allocate new memory for identical strings?
  5. Python 32-bit memory limits on 64bit windows
  6. Python out of memory on large CSV file (numpy)
  7. Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory”
  8. In-memory size of a Python structure
  9. What is the recommended way of allocating memory for a typed memory view?
  10. Memory errors and list limits?
  11. Access memory address in python
  12. How unique is Python’s id()?
  13. How to increase Jupyter notebook Memory limit?
  14. Cross-platform space remaining on volume using python
  15. Get total physical memory in Python
  16. Tracking *maximum* memory usage by a Python function
  17. Extracting a zipfile to memory?
  18. How to clear memory completely of all matplotlib plots
  19. Measure Object Size Accurately in Python – Sys.GetSizeOf not functioning
  20. Deep version of sys.getsizeof [duplicate]
  21. Python in-memory zip library
  22. Why do ints require three times as much memory in Python?
  23. When to use == and when to use is?
  24. How can I check the memory usage of objects in iPython?
  25. Python deep getsizeof list with contents?
  26. How to get largest possible precision? (Python – Decimal)
  27. Tensorflow Allocation Memory: Allocation of 38535168 exceeds 10% of system memory
  28. Limit RAM usage to python program
  29. Why is lxml.etree.iterparse() eating up all my memory?
  30. Why Python `Memory Error` with list `append()` lots of RAM left

Leave a Comment