Why doesn’t Python’s mmap work with large files?

by

From IEEE 1003.1:

The mmap() function shall establish a
mapping between a process’ address
space and a file, shared memory
object, or [TYM] typed memory
object.

It needs all the virtual address space because that’s exactly what mmap() does.

The fact that it isn’t really running out of memory doesn’t matter – you can’t map more address space than you have available. Since you then take the result and access as if it were memory, how exactly do you propose to access more than 2^32 bytes into the file? Even if mmap() didn’t fail, you could still only read the first 4GB before you ran out of space in a 32-bit address space. You can, of course, mmap() a sliding 32-bit window over the file, but that won’t necessarily net you any benefit unless you can optimize your access pattern such that you limit how many times you have to visit previous windows.

More Related Contents:

  1. Python readlines() usage and efficient practice for reading
  2. High Memory Usage Using Python Multiprocessing
  3. Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?
  4. How do I profile a Python script?
  5. String formatting: % vs. .format vs. f-string literal
  6. When should I (not) want to use pandas apply() in my code?
  7. How to measure elapsed time in Python?
  8. What is the difference between contiguous and non-contiguous arrays?
  9. Link ATLAS/MKL to an installed Numpy
  10. Fastest way to grow a numpy numeric array
  11. What is the fastest way to parse large XML docs in Python?
  12. What is the time complexity of popping elements from list in Python?
  13. Interactive large plot with ~20 million sample points and gigabytes of data
  14. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  15. Fast arbitrary distribution random sampling (inverse transform sampling)
  16. Mass string replace in python?
  17. Python: get a dict from a list based on something inside the dict
  18. Why is variable1 += variable2 much faster than variable1 = variable1 + variable2?
  19. Why is `np.sum(range(N))` very slow?
  20. Python built-in sum function vs. for loop performance
  21. Is freeing handled differently for small/large numpy arrays?
  22. Are NumPy’s math functions faster than Python’s?
  23. Creating an empty list in Python
  24. Python: iterating over list vs over dict items efficiency
  25. First items in inner list efficiently as possible [duplicate]
  26. Select the max row per group – pandas performance issue
  27. Does python logging flush every log?
  28. Django: Increment blog entry view count by one. Is this efficient?
  29. Creating a numpy array of 3D coordinates from three 1D arrays
  30. memory error in python

Leave a Comment