How do I assess the hash collision probability?

by

Equal hash means equal file, unless someone malicious is messing around with your files and injecting collisions. (this could be the case if they are downloading stuff from the internet) If that is the case go for a SHA2 based function.

There are no accidental MD5 collisions, 1,47×10-29 is a really really really small number.

To overcome the issue of rehashing big files I would have a 3 phased identity scheme.

  1. Filesize alone
  2. Filesize + a hash of 64K * 4 in different positions in the file
  3. A full hash

So if you see a file with a new size you know for certain you do not have a duplicate. And so on.

More Related Contents:

  1. What is the difference between a deep copy and a shallow copy?
  2. Should a function have only one return statement?
  3. What is the difference between concurrency and parallelism?
  4. Can every recursion be converted into iteration?
  5. Why shouldn’t I use “Hungarian Notation”?
  6. Expression Versus Statement
  7. How does XOR variable swapping work?
  8. Prefer composition over inheritance?
  9. Data structures for loaded dice?
  10. Singletons: good design or a crutch? [closed]
  11. Test loops at the top or bottom? (while vs. do while) [closed]
  12. Which, and why, do you prefer Exceptions or Return codes?
  13. Buffered vs unbuffered IO
  14. What is the difference between Type and Class?
  15. Allen Holub wrote “You should never use get/set functions”, is he correct? [duplicate]
  16. What are the lesser known but useful data structures?
  17. Semantic Diff Utilities [closed]
  18. Why is it recommended to have empty line in the end of a source file?
  19. What’s the best name for a non-mutating “add” method on an immutable collection? [closed]
  20. Are exceptions really for exceptional errors? [closed]
  21. What is a stack overflow?
  22. Generate colors between red and green for a power meter?
  23. Calculating a 2D Vector’s Cross Product
  24. Hash Code and Checksum – what’s the difference?
  25. How to name variables
  26. In what situations is octal base used?
  27. When should a method be static?
  28. How to design extensible software (plugin architecture)? [closed]
  29. XOR of three values
  30. String Compare “Logic”

Leave a Comment