Unable to allocate array with shape and data type

by

This is likely due to your system’s overcommit handling mode.

In the default mode, 0,

Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. The root is allowed to allocate slightly more memory in this mode. This is the default.

The exact heuristic used is not well explained here, but this is discussed more on Linux over commit heuristic and on this page.

You can check your current overcommit mode by running

$ cat /proc/sys/vm/overcommit_memory
0

In this case, you’re allocating

>>> 156816 * 36 * 53806 / 1024.0**3
282.8939827680588

~282 GB and the kernel is saying well obviously there’s no way I’m going to be able to commit that many physical pages to this, and it refuses the allocation.

If (as root) you run:

$ echo 1 > /proc/sys/vm/overcommit_memory

This will enable the “always overcommit” mode, and you’ll find that indeed the system will allow you to make the allocation no matter how large it is (within 64-bit memory addressing at least).

I tested this myself on a machine with 32 GB of RAM. With overcommit mode 0 I also got a MemoryError, but after changing it back to 1 it works:

>>> import numpy as np
>>> a = np.zeros((156816, 36, 53806), dtype="uint8")
>>> a.nbytes
303755101056

You can then go ahead and write to any location within the array, and the system will only allocate physical pages when you explicitly write to that page. So you can use this, with care, for sparse arrays.

More Related Contents:

  1. value error in python script
  2. Pandas conditional creation of a series/dataframe column
  3. How do I get indices of N maximum values in a NumPy array?
  4. Slice 2d array into smaller 2d arrays
  5. Pandas read_csv low_memory and dtype options
  6. How to pretty-print a numpy.array without scientific notation and with given precision?
  7. replace() method not working on Pandas DataFrame
  8. How to apply a disc shaped mask to a NumPy array?
  9. using an numpy array as indices of the 2nd dim of another array? [duplicate]
  10. How to evaluate the sum of values within array blocks
  11. Multiple linear regression in Python
  12. Numpy – create matrix with rows of vector
  13. Numpy integer nan [duplicate]
  14. Take multiple lists into dataframe
  15. Using Numpy Vectorize on Functions that Return Vectors
  16. Convert numpy.datetime64 to string object in python
  17. Matplotlib: how to make imshow read x,y coordinates from other numpy arrays?
  18. Numpy individual element access slower than for lists
  19. Why using an array as an index changes the shape of a multidimensional ndarray?
  20. How to get stable results with TensorFlow, setting random seed
  21. How to multiply individual elements of a list with a number?
  22. Numpy isnan() fails on an array of floats (from pandas dataframe apply)
  23. Installing lapack for numpy
  24. Adding colors to a 3d quiver plot in matplotlib
  25. Load CSV to Pandas MultiIndex DataFrame
  26. How to hash a large object (dataset) in Python?
  27. Python pandas – new column’s value if the item is in the list
  28. How to create a legend for 3D bar in matplotlib?
  29. Digitizing an analog signal
  30. calculate distance of 2 list of points in numpy

Leave a Comment