Get MD5 hash of big files in Python

by

You need to read the file in chunks of suitable size:

def md5_for_file(f, block_size=2**20):
    md5 = hashlib.md5()
    while True:
        data = f.read(block_size)
        if not data:
            break
        md5.update(data)
    return md5.digest()

NOTE: Make sure you open your file with the ‘rb’ to the open – otherwise you will get the wrong result.

So to do the whole lot in one method – use something like: 

def generate_file_md5(rootdir, filename, blocksize=2**20):
    m = hashlib.md5()
    with open( os.path.join(rootdir, filename) , "rb" ) as f:
        while True:
            buf = f.read(blocksize)
            if not buf:
                break
            m.update( buf )
    return m.hexdigest()

The update above was based on the comments provided by Frerich Raabe – and I tested this and found it to be correct on my Python 2.7.2 windows installation

I cross-checked the results using the ‘jacksum’ tool.

jacksum -a md5 <filename>

http://www.jonelo.de/java/jacksum/

More Related Contents:

  1. Generating an MD5 checksum of a file
  2. Hashing a file in Python
  3. Get the MD5 hash of big files in Python
  4. How to correct TypeError: Unicode-objects must be encoded before hashing?
  5. Error importing hashlib with python 2.7 but not with 2.6
  6. How to get MD5 sum of a string using python?
  7. How do I calculate the MD5 checksum of a file in Python? [duplicate]
  8. ‘frozenset’ object is not callable
  9. How can I safely create a nested directory?
  10. Explanation of how nested list comprehension works?
  11. Why does Python code run faster in a function?
  12. What’s the purpose of “send” function on Python generators?
  13. How to construct a timedelta object from a simple string
  14. How I can I lazily read multiple JSON values from a file/stream in Python?
  15. python: can executable zip files include data files?
  16. Python string ‘join’ is faster (?) than ‘+’, but what’s wrong here?
  17. Record speakers output with PyAudio
  18. How do I get an event callback when a Tkinter Entry widget is modified?
  19. How can I know which exceptions might be thrown from a method call?
  20. What is the purpose of the colon before a block in Python?
  21. Compare XML snippets?
  22. Tkinter – Geometry management
  23. Calculate summary statistics of columns in dataframe
  24. Pickle with custom classes
  25. How to convert an image from np.uint16 to np.uint8?
  26. pandas plot value counts barplot in descending manner [duplicate]
  27. How to calculate length of string in pixels for specific font and size?
  28. Get the last day of the month
  29. How can I avoid issues caused by Python’s early-bound default parameters (e.g. mutable default arguments “remembering” old data)?
  30. What is the Spark DataFrame method `toPandas` actually doing?

Leave a Comment