Memory error due to the huge input file size

by

Obviously the file is too large to be read into memory all at once.

Why not just use:

with open("data.txt") as myfile:
    for line in myfile:
        do_something(line.rstrip("\n"))

or, if you’re not on Python 2.6 and higher:

myfile = open("data.txt")
for line in myfile:
    do_something(line.rstrip("\n"))

In both cases, you’ll get an iterator that can be treated much like a list of strings.

EDIT: Since your way of reading the entire file into one large string and then splitting it on newlines will remove the newlines in the process, I have added a .rstrip("\n") to my examples in order to better simulate the result.

More Related Contents:

  1. How to populate Objects in a an existing file [closed]
  2. Turn the whole file uppercase
  3. Is explicitly closing files important?
  4. Get last n lines of a file, similar to tail
  5. Iterating on a file doesn’t work the second time
  6. Directory-tree listing in Python
  7. How to write to a CSV line by line?
  8. Open file in a relative location in Python
  9. Read and overwrite a file in Python
  10. check what files are open in Python
  11. How to implement a pythonic equivalent of tail -F?
  12. How to convert a file to utf-8 in Python?
  13. replacing text in a file with Python
  14. Read from a log file as it’s being written using python
  15. How do I call a function from another .py file?
  16. Is close() necessary when using iterator on a Python file object [duplicate]
  17. Finding duplicate files and removing them
  18. Compare two files report difference in python
  19. os.walk without digging into directories below
  20. Python 2.7: Print to File
  21. Create a temporary FIFO (named pipe) in Python?
  22. What is the idiomatic way to iterate over a binary file?
  23. Read from File, or STDIN
  24. Detect duplicate MP3 files with different bitrates and/or different ID3 tags?
  25. Error with double backslash in Windows path in Python [duplicate]
  26. Sending a file over TCP sockets in Python
  27. Is there a way to efficiently yield every file in a directory containing millions of files?
  28. Read random lines from huge CSV file
  29. What is the difference between .py and .pyc files? [duplicate]
  30. how to count the total number of lines in a text file using python

Leave a Comment