Process very large (>20GB) text file line by line

by

It’s more idiomatic to write your code like this

def ProcessLargeTextFile():
    with open("filepath", "r") as r, open("outfilepath", "w") as w:
        for line in r:
            x, y, z = line.split(' ')[:3]
            w.write(line.replace(x,x[:-3]).replace(y,y[:-3]).replace(z,z[:-3]))

The main saving here is to just do the split once, but if the CPU is not being taxed, this is likely to make very little difference

It may help to save up a few thousand lines at a time and write them in one hit to reduce thrashing of your harddrive. A million lines is only 54MB of RAM!

def ProcessLargeTextFile():
    bunchsize = 1000000     # Experiment with different sizes
    bunch = []
    with open("filepath", "r") as r, open("outfilepath", "w") as w:
        for line in r:
            x, y, z = line.split(' ')[:3]
            bunch.append(line.replace(x,x[:-3]).replace(y,y[:-3]).replace(z,z[:-3]))
            if len(bunch) == bunchsize:
                w.writelines(bunch)
                bunch = []
        w.writelines(bunch)

suggested by @Janne, an alternative way to generate the lines

def ProcessLargeTextFile():
    bunchsize = 1000000     # Experiment with different sizes
    bunch = []
    with open("filepath", "r") as r, open("outfilepath", "w") as w:
        for line in r:
            x, y, z, rest = line.split(' ', 3)
            bunch.append(' '.join((x[:-3], y[:-3], z[:-3], rest)))
            if len(bunch) == bunchsize:
                w.writelines(bunch)
                bunch = []
        w.writelines(bunch)

More Related Contents:

  1. Printing the number from a list and its square on the same line in python3 [closed]
  2. How to read specific lines from a file (by line number)?
  3. How do I compute the intersection point of two lines?
  4. Python OpenCV line detection to detect `X` symbol in image
  5. Matplotlib: Plotting numerous disconnected line segments with different colors
  6. Delete final line in file with python
  7. Read lines from one file and write to another file but remove lines that contain certain string
  8. How can I format a list to print each element on a separate line in python? [duplicate]
  9. matplotlib 2d line line,=plot comma meaning
  10. pip install from git repo branch
  11. How to manage local vs production settings in Django?
  12. where to put freeze_support() in a Python script?
  13. Python exception chaining [duplicate]
  14. Threads and tkinter
  15. Equivalent of Bash Backticks in Python [duplicate]
  16. How do I reverse a part (slice) of a list in Python?
  17. Save Dataframe to csv directly to s3 Python
  18. How do you uninstall a python package that was installed using distutils?
  19. Editing the date formatting of x-axis tick labels
  20. How to split an integer into a list of digits?
  21. Why doesn’t assigning to the loop variable modify the original list? How can I assign back to the list in a loop? [duplicate]
  22. Playing note with pygame.midi
  23. parsing a fasta file using a generator ( python )
  24. Big Sur clang “invalid version” error due to MACOSX_DEPLOYMENT_TARGET
  25. Numpy difference between neighboring elements
  26. Pandas: parse merged header columns from Excel
  27. new pythonic style for shared axes square subplots in matplotlib?
  28. Catching exception in context manager __enter__()
  29. Beautiful Soup and Table Scraping – lxml vs html parser
  30. How to install multiple python packages at once using pip

Leave a Comment