How might I remove duplicate lines from a file?

by

On Unix/Linux, use the uniq command, as per David Locke’s answer, or sort, as per William Pursell’s comment.

If you need a Python script:

lines_seen = set() # holds lines already seen
outfile = open(outfilename, "w")
for line in open(infilename, "r"):
    if line not in lines_seen: # not a duplicate
        outfile.write(line)
        lines_seen.add(line)
outfile.close()

Update: The sort/uniq combination will remove duplicates but return a file with the lines sorted, which may or may not be what you want. The Python script above won’t reorder lines, but just drop duplicates. Of course, to get the script above to sort as well, just leave out the outfile.write(line) and instead, immediately after the loop, do outfile.writelines(sorted(lines_seen)).

More Related Contents:

  1. Print string to text file
  2. How do I delete a file or folder in Python?
  3. How do I concatenate text files in Python?
  4. Extracting text from HTML file using Python
  5. Get last n lines of a file, similar to tail
  6. Read .mat files in Python
  7. Create Pandas DataFrame from txt file with specific pattern
  8. How to bind self events in Tkinter Text widget after it will binded by Text widget?
  9. How to split text without spaces into list of words
  10. Reading from a frequently updated file
  11. open() in Python does not create a file if it doesn’t exist
  12. Does reading an entire file leave the file handle open?
  13. How to make file creation an atomic operation?
  14. How to delete specific strings from a file?
  15. Replace string within file contents [duplicate]
  16. Batch Renaming of Files in a Directory
  17. PIL – draw multiline text on image
  18. How to easily print ascii-art text? [closed]
  19. How to count the number of words in a sentence, ignoring numbers, punctuation and whitespace?
  20. Efficiently finding the last line in a text file [duplicate]
  21. Python Random Access File
  22. How can I read large text files line by line, without loading them into memory? [duplicate]
  23. How can I print bold text in Python?
  24. Parsing apache log files
  25. Trying to use open(filename, ‘w’ ) gives IOError: [Errno 2] No such file or directory if directory doesn’t exist
  26. Python file operations
  27. Speed up writing to files
  28. Treat binary data as a file object?
  29. Unicode encoding for filesystem in Mac OS X not correct in Python?
  30. Mixing read() and write() on Python files in Windows

Leave a Comment