Python Slice Assignment Memory Usage

by

The line

a[:] = [i + 6 for i in a]

would not save any memory. Python does evaluate the right hand side first, as stated in the language documentation:

An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple) and assigns the single resulting object to each of the target lists, from left to right.

In the case at hand, the single resulting object would be a new list, and the single target in the target list would be a[:].

We could replace the list comprehension by a generator expression:

a[:] = (i + 6 for i in a)

Now, the right hand side evaluates to a generator instead of a list. Benchmarking shows that this is still slower than the naive

a = [i + 6 for i in a]

So does the generator expression actually save any memory? At first glance, you might think it does. But delving in to the source code of the function list_ass_slice() shows that it does not. The line

v_as_SF = PySequence_Fast(v, "can only assign an iterable");

uses PySequence_Fast() to convert the iterable (in this case the generator) into a tuple first, which is then copied into the old list. A tuple uses the same amount of memory as a list, so using a generator expression is basically the same as using a list comprehension in this case. During the last copy, the items of the original list are reused.

The moral seems to be that the simplest approach is the best one in any regard.

More Related Contents:

  1. Which Python memory profiler is recommended? [closed]
  2. Why does Python code run faster in a function?
  3. Why is looping over range() in Python faster than using a while loop?
  4. python: deque vs list performance comparison
  5. Python Requests vs PyCurl Performance
  6. Python Numpy Data Types Performance
  7. Is doing multiplication multiple times or assigning to a variable faster?
  8. How can I explicitly free memory in Python?
  9. pandas loc vs. iloc vs. at vs. iat?
  10. What is the best way to generate all possible three letter strings?
  11. Why is [] faster than list()?
  12. Why is numpy’s einsum faster than numpy’s built in functions?
  13. Speed comparison with Project Euler: C vs Python vs Erlang vs Haskell
  14. What’s a faster operation, re.match/search or str.find?
  15. Cost of exception handlers in Python
  16. File read using “open()” vs “with open()” [duplicate]
  17. It is more efficient to use if-return-return or if-else-return?
  18. Most efficient way to forward-fill NaN values in numpy array
  19. [] and {} vs list() and dict(), which is better?
  20. How do I get monotonic time durations in python?
  21. multiprocessing.Pool() slower than just using ordinary functions
  22. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  23. Complexity of list.index(x) in Python
  24. What do I need K.clear_session() and del model for (Keras with Tensorflow-gpu)?
  25. How do I check if PyTorch is using the GPU?
  26. What is the fastest way to output large DataFrame into a CSV file?
  27. How expensive are Python dictionaries to handle?
  28. What is the most efficient way to check if a value exists in a NumPy array?
  29. What is the most efficient way of doing square root of sum of square of two numbers?
  30. Python if vs try-except

Leave a Comment