How to convert an iterable to a stream?

by

Python 3 has a new I/O stream API (library docs), replacing the old file-like object protocol. (The new API is also available in Python 2 in the io module, and it’s backwards-compatible with the file-like object protocol.)

Here’s an implementation for the new API, in Python 2 and 3:

import io

def iterable_to_stream(iterable, buffer_size=io.DEFAULT_BUFFER_SIZE):
    """
    Lets you use an iterable (e.g. a generator) that yields bytestrings as a read-only
    input stream.

    The stream implements Python 3's newer I/O API (available in Python 2's io module).
    For efficiency, the stream is buffered.
    """
    class IterStream(io.RawIOBase):
        def __init__(self):
            self.leftover = None
        def readable(self):
            return True
        def readinto(self, b):
            try:
                l = len(b)  # We're supposed to return at most this much
                chunk = self.leftover or next(iterable)
                output, self.leftover = chunk[:l], chunk[l:]
                b[:len(output)] = output
                return len(output)
            except StopIteration:
                return 0    # indicate EOF
    return io.BufferedReader(IterStream(), buffer_size=buffer_size)

Example usage:

with iterable_to_stream(str(x**2).encode('utf8') for x in range(11)) as s:
    print(s.read())

More Related Contents:

  1. Download large file in python with requests
  2. How to build a basic iterator?
  3. How can I iterate over files in a given directory?
  4. Get the first item from an iterable that matches a condition
  5. Difference between Python’s Generators and Iterators
  6. Iterate over all pairs of consecutive items in a list [duplicate]
  7. Circular list iterator in Python
  8. Can iterators be reset in Python?
  9. What’s the shortest way to count the number of items in a generator/iterator?
  10. If range() is a generator in Python 3.3, why can I not call next() on a range?
  11. Is close() necessary when using iterator on a Python file object [duplicate]
  12. Is there any built-in way to get the length of an iterable in python?
  13. Is there an expression for an infinite iterator?
  14. What does “three dots” in Python mean when indexing what looks like a number?
  15. When is not a good time to use python generators?
  16. Making a python iterator go backwards?
  17. split a generator/iterable every n items in python (splitEvery)
  18. Can you “stream” images to ffmpeg to construct a video, instead of saving them to disk?
  19. Iterate over the lines of a string
  20. Why does next raise a ‘StopIteration’, but ‘for’ do a normal return?
  21. Performance Advantages to Iterators?
  22. Python range() and zip() object type
  23. Fastest way to convert an iterator to a list
  24. How can I iterate twice over a given iterator (reuse the data, “reset” the iterator etc.)? Why does iterating twice give a blank result?
  25. What is the idiomatic way to iterate over a binary file?
  26. One-liner to check whether an iterator yields at least one element?
  27. Confused with python lists: are they or are they not iterators? [duplicate]
  28. How to read the first byte of a subprocess’s stdout and then discard the rest in Python?
  29. What does it mean to “consume” in Python? In an iterator?
  30. hasnext() for Python iterators?

Leave a Comment