Memory errors and list limits?

by

First off, see How Big can a Python Array Get? and Numpy, problem with long arrays

Second, the only real limit comes from the amount of memory you have and how your system stores memory references. There is no per-list limit, so Python will go until it runs out of memory. Two possibilities:

  1. If you are running on an older OS or one that forces processes to use a limited amount of memory, you may need to increase the amount of memory the Python process has access to.
  2. Break the list apart using chunking. For example, do the first 1000 elements of the list, pickle and save them to disk, and then do the next 1000. To work with them, unpickle one chunk at a time so that you don’t run out of memory. This is essentially the same technique that databases use to work with more data than will fit in RAM.

More Related Contents:

  1. Why Python `Memory Error` with list `append()` lots of RAM left
  2. How to iterate through two lists in parallel?
  3. Transpose/Unzip Function (inverse of zip)?
  4. Are lists thread-safe?
  5. Python List vs. Array – when to use?
  6. Getting the index of the returned max or min item using max()/min() on a list
  7. How can I iterate through two lists in parallel?
  8. Find and replace string values in list
  9. Pairs from single list
  10. Splitting a list based on a delimiter word
  11. How do you pick “x” number of unique numbers from a list in Python?
  12. Naming conflict with built-in function
  13. How do I split a list into equally-sized chunks?
  14. How can I make a dictionary from separate lists of keys and values?
  15. Confused about __str__ on list in Python [duplicate]
  16. join two lists of dictionaries on a single key
  17. Python – How to sort a list of lists by the fourth element in each list? [duplicate]
  18. Confusing […] List in Python: What is it?
  19. How to delete an item in a list if it exists?
  20. Flattening a list recursively [duplicate]
  21. remove None value from a list without removing the 0 value
  22. Python: Remove Duplicate Items from Nested list
  23. Changing one dict changes all dicts in a list of dicts [duplicate]
  24. Python string.join(list) on object array rather than string array
  25. How to sort multidimensional array by column?
  26. sum each value in a list of tuples
  27. How to combine elements from two lists into a third?
  28. smartest way to join two lists into a formatted string
  29. Sum list of list elements in python like sql group by [duplicate]
  30. Python : how to append new elements in a list of list?

Leave a Comment