Lazy Method for Reading Big File in Python?

by

To write a lazy function, just use yield:

def read_in_chunks(file_object, chunk_size=1024):
    """Lazy function (generator) to read a file piece by piece.
    Default chunk size: 1k."""
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data


with open('really_big_file.dat') as f:
    for piece in read_in_chunks(f):
        process_data(piece)

Another option would be to use iter and a helper function:

f = open('really_big_file.dat')
def read1k():
    return f.read(1024)

for piece in iter(read1k, ''):
    process_data(piece)

If the file is line-based, the file object is already a lazy generator of lines:

for line in open('really_big_file.dat'):
    process_data(line)

More Related Contents:

  1. Reading from a frequently updated file
  2. Create or open file in python [closed]
  3. Correct way to write line to file?
  4. Generator expressions vs. list comprehensions
  5. Print string to text file
  6. Python recursive folder read
  7. how to split an iterable in constant-size chunks
  8. Is there a generator version of `string.split()` in Python?
  9. Python random sample with a generator / iterable / iterator
  10. Difference between parsing a text file in r and rb mode
  11. How to look ahead one element (peek) in a Python generator?
  12. Is there an expression for an infinite iterator?
  13. Inline CSV File Editing with Python
  14. When is not a good time to use python generators?
  15. How to check if an object is a generator object in python?
  16. python 3 print generator
  17. Python not recognising directories os.path.isdir() [duplicate]
  18. What is the purpose of the “send” function on Python generators?
  19. Python read() function returns empty string [closed]
  20. Is pickle file of python cross-platform?
  21. How can I delete a file or folder in Python?
  22. What is the difference between rb and r+b modes in file objects [duplicate]
  23. Write file with specific permissions in Python
  24. How to create the union of many sets using a generator expression?
  25. Saving response from Requests to file
  26. How to insert / retrieve a file stored as a BLOB in a MySQL db using python
  27. Do python’s variable length arguments (*args) expand a generator at function call time?
  28. Python – How to check if a file is used by another application?
  29. Coroutine vs Continuation vs Generator
  30. how to count the total number of lines in a text file using python

Leave a Comment