Iterate an iterator by chunks (of n) in Python?

by

The grouper() recipe from the itertools documentation’s recipes comes close to what you want:

def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

It will fill up the last chunk with a fill value, though.

A less general solution that only works on sequences but does handle the last chunk as desired is

[my_list[i:i + chunk_size] for i in range(0, len(my_list), chunk_size)]

Finally, a solution that works on general iterators and behaves as desired is

def grouper(n, iterable):
    it = iter(iterable)
    while True:
        chunk = tuple(itertools.islice(it, n))
        if not chunk:
            return
        yield chunk

More Related Contents:

  1. How to iterate through two lists in parallel?
  2. What exactly are iterator, iterable, and iteration?
  3. How to build a basic iterator?
  4. How can I iterate over files in a given directory?
  5. Iterate over all pairs of consecutive items in a list [duplicate]
  6. Circular list iterator in Python
  7. Getting number of elements in an iterator in Python
  8. How to read file N lines at a time?
  9. Can iterators be reset in Python?
  10. What’s the shortest way to count the number of items in a generator/iterator?
  11. How do I iterate through two lists in parallel?
  12. Is there any built-in way to get the length of an iterable in python?
  13. Skip multiple iterations in loop
  14. Is there an expression for an infinite iterator?
  15. What does “three dots” in Python mean when indexing what looks like a number?
  16. When is not a good time to use python generators?
  17. How to convert an iterable to a stream?
  18. Making a python iterator go backwards?
  19. python groupby behaviour?
  20. split a generator/iterable every n items in python (splitEvery)
  21. cleanest way to call one function on a list of items
  22. Iterate over the lines of a string
  23. Performance Advantages to Iterators?
  24. has_next in Python iterators?
  25. Why can’t I iterate twice over the same iterator? How can I “reset” the iterator or reuse the data?
  26. Proper way to reset csv.reader for multiple iterations?
  27. One-liner to check whether an iterator yields at least one element?
  28. Confused with python lists: are they or are they not iterators? [duplicate]
  29. What does it mean to “consume” in Python? In an iterator?
  30. hasnext() for Python iterators?

Leave a Comment