List vs generator comprehension speed with join function [duplicate]

by

The str.join method converts its iterable parameter to a list if it’s not a list or tuple already. This lets the joining logic iterate over the items multiple times (it makes one pass to calculate the size of the result string, then a second pass to actually copy the data).

You can see this in the CPython source code:

PyObject *
PyUnicode_Join(PyObject *separator, PyObject *seq)
{
    /* lots of variable declarations at the start of the function omitted */

    fseq = PySequence_Fast(seq, "can only join an iterable");

    /* ... */
}

The PySequence_Fast function in the C API does just what I described. It converts an arbitrary iterable into a list (essentially by calling list on it), unless it already is a list or tuple.

The conversion of the generator expression to a list means that the usual benefits of generators (a smaller memory footprint and the potential for short-circuiting) don’t apply to str.join, and so the (small) additional overhead that the generator has makes its performance worse.

More Related Contents:

  1. Why is a list comprehension so much faster than appending to a list?
  2. Why do list comprehensions write to the loop variable, but generators don’t? [duplicate]
  3. sum each value in a list of tuples
  4. flatten list of list through list comprehension
  5. range(len(list)) or enumerate(list)? [duplicate]
  6. Skip elements on a condition based in a list comprehension in python
  7. How does all() in python work on empty lists
  8. How to match a string to pattern “Foo-Bar”, where Bar can be any element of a list? [closed]
  9. Check if the element is last in the list
  10. What’s wrong with my Python 2 program?
  11. Create a dictionary with comprehension
  12. How do you send a HEAD HTTP request in Python 2?
  13. recursive iteration through nested json for specific key in python
  14. How would I stop a while loop after n amount of time?
  15. How to extract text from an existing docx file using python-docx
  16. ImportError: No module named _ssl
  17. list() uses slightly more memory than list comprehension
  18. How to recursively find specific key in nested JSON?
  19. Chromedriver, Selenium – Automate downloads
  20. How to get Windows short file name in python?
  21. What does the “yield” keyword do in Python?
  22. Get a unique list of items that occur more than once in a list
  23. Name is not defined in a list comprehension with multiple loops
  24. Python: setting two variable values separated by a comma in python
  25. Switch between python 2.7 and python 3.5 on Mac OS X
  26. How to modify the default font in Tkinter?
  27. creating a spiral array in python?
  28. Python returns length of 2 for single Unicode character string
  29. Why do I get “‘str’ object has no attribute ‘read'” when trying to use `json.load` on a string? [duplicate]
  30. How to handle IncompleteRead: in python

Leave a Comment