Reading a huge .csv file

by

You are reading all rows into a list, then processing that list. Don’t do that.

Process your rows as you produce them. If you need to filter the data first, use a generator function:

import csv

def getstuff(filename, criterion):
    with open(filename, "rb") as csvfile:
        datareader = csv.reader(csvfile)
        yield next(datareader)  # yield the header row
        count = 0
        for row in datareader:
            if row[3] == criterion:
                yield row
                count += 1
            elif count:
                # done when having read a consecutive series of rows 
                return

I also simplified your filter test; the logic is the same but more concise.

Because you are only matching a single sequence of rows matching the criterion, you could also use:

import csv
from itertools import dropwhile, takewhile

def getstuff(filename, criterion):
    with open(filename, "rb") as csvfile:
        datareader = csv.reader(csvfile)
        yield next(datareader)  # yield the header row
        # first row, plus any subsequent rows that match, then stop
        # reading altogether
        # Python 2: use `for row in takewhile(...): yield row` instead
        # instead of `yield from takewhile(...)`.
        yield from takewhile(
            lambda r: r[3] == criterion,
            dropwhile(lambda r: r[3] != criterion, datareader))
        return

You can now loop over getstuff() directly. Do the same in getdata():

def getdata(filename, criteria):
    for criterion in criteria:
        for row in getstuff(filename, criterion):
            yield row

Now loop directly over getdata() in your code:

for row in getdata(somefilename, sequence_of_criteria):
    # process row

You now only hold one row in memory, instead of your thousands of lines per criterion.

yield makes a function a generator function, which means it won’t do any work until you start looping over it.

More Related Contents:

  1. Writing a Python list of lists to a csv file
  2. How do I create a python code that distinguishes between the capital of a state and the state that the capital has?
  3. Skip the headers when editing a csv file using Python
  4. How to write to a CSV line by line?
  5. Read and Write CSV files including unicode with Python 2.7
  6. Python convert csv to xlsx
  7. Python parse CSV ignoring comma with double-quotes
  8. file.tell() inconsistency
  9. portable way to write csv file in python 2 or python 3
  10. Use Multiple Character Delimiter in Python Pandas read_csv
  11. Fastest way to process a large file?
  12. Python 2.7: Print to File
  13. Is there a need to close files that have no reference to them?
  14. How to read one single line of csv data in Python?
  15. Mixing files and loops
  16. How can I use io.StringIO() with the csv module?
  17. Sending a file over TCP sockets in Python
  18. How to read the last line of a file in Python? [duplicate]
  19. Python Pandas – Read csv file containing multiple tables
  20. Find all CSV files in a directory using Python
  21. Read random lines from huge CSV file
  22. How to count top 10 most common values in a dict in python
  23. Python not able to open file with non-english characters in path
  24. Get data from Twitter using Tweepy and store in csv file
  25. Understanding recursion in Python
  26. Most efficient way to search the last X lines of a file?
  27. selecting across multiple columns with python pandas?
  28. Not able to parse a .csv file uploaded using Flask
  29. How can I do a batch insert into an Oracle database using Python?
  30. What are the differences between bool() and operator.truth()?

Leave a Comment