See if two files have the same content in python [duplicate]

by

Yes, I think hashing the file would be the best way if you have to compare several files and store hashes for later comparison. As hash can clash, a byte-by-byte comparison may be done depending on the use case.

Generally byte-by-byte comparison would be sufficient and efficient, which filecmp module already does + other things too.

See http://docs.python.org/library/filecmp.html
e.g.

>>> import filecmp
>>> filecmp.cmp('file1.txt', 'file1.txt')
True
>>> filecmp.cmp('file1.txt', 'file2.txt')
False

Speed consideration:
Usually if only two files have to be compared, hashing them and comparing them would be slower instead of simple byte-by-byte comparison if done efficiently. e.g. code below tries to time hash vs byte-by-byte

Disclaimer: this is not the best way of timing or comparing two algo. and there is need for improvements but it does give rough idea. If you think it should be improved do tell me I will change it.

import random
import string
import hashlib
import time

def getRandText(N):
    return  "".join([random.choice(string.printable) for i in xrange(N)])

N=1000000
randText1 = getRandText(N)
randText2 = getRandText(N)

def cmpHash(text1, text2):
    hash1 = hashlib.md5()
    hash1.update(text1)
    hash1 = hash1.hexdigest()

    hash2 = hashlib.md5()
    hash2.update(text2)
    hash2 = hash2.hexdigest()

    return  hash1 == hash2

def cmpByteByByte(text1, text2):
    return text1 == text2

for cmpFunc in (cmpHash, cmpByteByByte):
    st = time.time()
    for i in range(10):
        cmpFunc(randText1, randText2)
    print cmpFunc.func_name,time.time()-st

and the output is

cmpHash 0.234999895096
cmpByteByByte 0.0

More Related Contents:

  1. Turn the whole file uppercase
  2. How do you append to a file?
  3. Is explicitly closing files important?
  4. How to modify a text file?
  5. Confused by python file mode “w+”
  6. Iterating on a file doesn’t work the second time
  7. Directory-tree listing in Python
  8. How to write to a CSV line by line?
  9. Importing variables from another file?
  10. How to delete the contents of a folder?
  11. Open file in a relative location in Python
  12. check what files are open in Python
  13. How to convert a file to utf-8 in Python?
  14. Read from a log file as it’s being written using python
  15. Is close() necessary when using iterator on a Python file object [duplicate]
  16. Finding duplicate files and removing them
  17. Compare two files report difference in python
  18. Python 2.7: Print to File
  19. Reading files in a particular order in python
  20. Python Random Access File
  21. Best method for reading newline delimited files and discarding the newlines?
  22. Create a temporary FIFO (named pipe) in Python?
  23. Sending a file over TCP sockets in Python
  24. UnicodeDecodeError in Python when reading a file, how to ignore the error and jump to the next line?
  25. Pydub (WindowsError: [Error 2] The system can not find the file specified)
  26. How can I extract the folder path from file path in Python?
  27. Use endswith with multiple extensions
  28. Creating files and directories via Python
  29. HTML to IMAGE using Python
  30. Change the file extension for files in a folder?

Leave a Comment