when does Python allocate new memory for identical strings?

by

Each implementation of the Python language is free to make its own tradeoffs in allocating immutable objects (such as strings) — either making a new one, or finding an existing equal one and using one more reference to it, are just fine from the language’s point of view. In practice, of course, real-world implementation strike reasonable compromise: one more reference to a suitable existing object when locating such an object is cheap and easy, just make a new object if the task of locating a suitable existing one (which may or may not exist) looks like it could potentially take a long time searching.

So, for example, multiple occurrences of the same string literal within a single function will (in all implementations I know of) use the “new reference to same object” strategy, because when building that function’s constants-pool it’s pretty fast and easy to avoid duplicates; but doing so across separate functions could potentially be a very time-consuming task, so real-world implementations either don’t do it at all, or only do it in some heuristically identified subset of cases where one can hope for a reasonable tradeoff of compilation time (slowed down by searching for identical existing constants) vs memory consumption (increased if new copies of constants keep being made).

I don’t know of any implementation of Python (or for that matter other languages with constant strings, such as Java) that takes the trouble of identifying possible duplicates (to reuse a single object via multiple references) when reading data from a file — it just doesn’t seem to be a promising tradeoff (and here you’d be paying runtime, not compile time, so the tradeoff is even less attractive). Of course, if you know (thanks to application level considerations) that such immutable objects are large and quite prone to many duplications, you can implement your own “constants-pool” strategy quite easily (intern can help you do it for strings, but it’s not hard to roll your own for, e.g., tuples with immutable items, huge long integers, and so forth).

More Related Contents:

  1. How can I explicitly free memory in Python?
  2. How do I determine the size of an object in Python?
  3. Does Python have a stack/heap and how is memory managed?
  4. What is the recommended way of allocating memory for a typed memory view?
  5. Is freeing handled differently for small/large numpy arrays?
  6. Which Python memory profiler is recommended? [closed]
  7. Total memory used by Python process?
  8. Is it possible to dereference variable id’s?
  9. What is the difference between contiguous and non-contiguous arrays?
  10. Python memory leaks [closed]
  11. How do I release memory used by a pandas dataframe?
  12. In-memory size of a Python structure
  13. What does “del” do exactly?
  14. Keep persistent variables in memory between runs of Python script
  15. How to limit the heap size?
  16. Upper memory limit?
  17. Python object deleting itself
  18. High Memory Usage Using Python Multiprocessing
  19. Is there a way to delete created variables, functions, etc from the memory of the interpreter?
  20. Best way to join two large datasets in Pandas
  21. Python/Numpy MemoryError
  22. How can I check the memory usage of objects in iPython?
  23. Techniques for working with large Numpy arrays? [duplicate]
  24. Confused about Python’s id() [duplicate]
  25. Python multiprocessing application is getting stuck in docker container
  26. How to clear variables in ipython?
  27. In Python, what is `sys.maxsize`?
  28. Limit Python VM memory
  29. Why doesn’t Python’s mmap work with large files?
  30. Create a figure that is reference counted

Leave a Comment