How to hash a large object (dataset) in Python?

by

Thanks to John Montgomery I think I have found a solution, and I think it has less overhead than converting every number in possibly huge arrays to strings:

I can create a byte-view of the arrays and use these to update the hash. And somehow this seems to give the same digest as directly updating using the array:

>>> import hashlib
>>> import numpy
>>> a = numpy.random.rand(10, 100)
>>> b = a.view(numpy.uint8)
>>> print a.dtype, b.dtype # a and b have a different data type
float64 uint8
>>> hashlib.sha1(a).hexdigest() # byte view sha1
'794de7b1316b38d989a9040e6e26b9256ca3b5eb'
>>> hashlib.sha1(b).hexdigest() # array sha1
'794de7b1316b38d989a9040e6e26b9256ca3b5eb'

More Related Contents:

  1. best way to preserve numpy arrays on disk
  2. Hashing a file in Python
  3. Preserve custom attributes when pickling subclass of numpy array
  4. ImportError: No module named ‘pandas.indexes’
  5. Python pickle protocol choice?
  6. ValueError: setting an array element with a sequence
  7. Numpy `logical_or` for more than two arguments
  8. Converting numpy dtypes to native python types
  9. Concatenate a NumPy array to another NumPy array
  10. ImportError: numpy.core.multiarray failed to import
  11. Fast replacement of values in a numpy array
  12. Link ATLAS/MKL to an installed Numpy
  13. Concatenate two NumPy arrays vertically
  14. Recursive definitions in Pandas
  15. numpy is already installed with Anaconda but I get an ImportError (DLL load failed: The specified module could not be found)
  16. Relationship between SciPy and NumPy
  17. How to specify upper and lower limits when using numpy.random.normal
  18. Are NumPy’s math functions faster than Python’s?
  19. NumPy chained comparison with two predicates
  20. Using datetime as ticks in Matplotlib
  21. How to count values in a certain range in a Numpy array?
  22. Python: why pickle?
  23. How to remove all rows in a numpy.ndarray that contain non-numeric values
  24. Python pandas: how to remove nan and -inf values
  25. Parse a Pandas column to Datetime when importing table from SQL database and filtering rows by date
  26. Dump a list in a pickle file and retrieve it back later [closed]
  27. What are some possible calculations with numpy or scipy that can return a NaN? [closed]
  28. Can’t import cv2; “DLL load failed”
  29. A numpy array unexpectedly changes when changing another one despite being separate
  30. Computing cross-correlation function?

Leave a Comment