How to remove duplicates from a csv file

by

UPDATE: 2016

If you are happy to use the helpful more_itertools external library:

from more_itertools import unique_everseen
with open('1.csv', 'r') as f, open('2.csv', 'w') as out_file:
    out_file.writelines(unique_everseen(f))

A more efficient version of @IcyFlame’s solution

with open('1.csv', 'r') as in_file, open('2.csv', 'w') as out_file:
    seen = set() # set for fast O(1) amortized lookup
    for line in in_file:
        if line in seen: continue # skip duplicate

        seen.add(line)
        out_file.write(line)

To edit the same file in-place you could use this (Old Python 2 code)

import fileinput
seen = set() # set for fast O(1) amortized lookup
for line in fileinput.FileInput('1.csv', inplace=1):
    if line in seen: continue # skip duplicate

    seen.add(line)
    print line, # standard output is now redirected to the file

More Related Contents:

  1. A simple maths operation on a CSV column using python [closed]
  2. How to write a python code which copies a unique column from an output file to a .csv file
  3. How can I convert JSON to CSV?
  4. Skip the headers when editing a csv file using Python
  5. How to add a new column to a CSV file?
  6. Skip rows during csv import pandas
  7. How to avoid Python/Pandas creating an index in a saved csv?
  8. datetime dtypes in pandas read_csv
  9. Python CSV error: line contains NULL byte
  10. How do I convert this list of dictionaries to a csv file?
  11. What is the fastest way to upload a big csv file in notebook to work with python pandas?
  12. sort csv by column
  13. csv.Error: iterator should return strings, not bytes
  14. Compare two CSV files and search for similar items
  15. Python Dictionary to CSV
  16. Is it possible to keep the column order using csv.DictReader?
  17. OSError: Initializing from file failed on csv in Pandas
  18. ValueError : I/O operation on closed file
  19. Writing to a particular cell using csv module in python
  20. Writing List of Strings to Excel CSV File in Python
  21. Load CSV file with PySpark
  22. Python UTF-16 CSV reader
  23. Read specific columns with pandas or other python module
  24. load csv into 2D matrix with numpy for plotting
  25. Read .csv file from URL into Python 3.x – _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
  26. Strip white spaces from CSV file
  27. Get data from Twitter using Tweepy and store in csv file
  28. How to add a new column to the beginning of the rows of a CSV file?
  29. Python Parse CSV Correctly
  30. start index at 1 for Pandas DataFrame

Leave a Comment