What you describe is called a collision. Collisions necessarily exist, since SHA-1 accepts many more distinct messages as input that it can produce distinct outputs (SHA-1 may eat any string of bits up to 2^64 bits, but outputs only 160 bits; thus, at least one output value must pop up several times). This observation is … Read more
using (FileStream fs = new FileStream(@”C:\file\location”, FileMode.Open)) using (BufferedStream bs = new BufferedStream(fs)) { using (SHA1Managed sha1 = new SHA1Managed()) { byte[] hash = sha1.ComputeHash(bs); StringBuilder formatted = new StringBuilder(2 * hash.Length); foreach (byte b in hash) { formatted.AppendFormat(“{0:X2}”, b); } } } formatted contains the string representation of the SHA-1 hash. Also, by using … Read more
The problem here is that SHA256Managed reads 4096 bytes at a time (inherit from FileStream and override Read(byte[], int, int) to see how much it reads from the filestream), which is too small a buffer for disk IO. To speed things up (2 minutes for hashing 2 Gb file on my machine with SHA256, 1 … Read more
This code is from the CMupdater, from the CyanogenMod 10.2 android ROM. It tests the downloaded ROMs into the updater App. code: https://github.com/CyanogenMod/android_packages_apps_CMUpdater/blob/cm-10.2/src/com/cyanogenmod/updater/utils/MD5.java It works like a charm: /* * Copyright (C) 2012 The CyanogenMod Project * * * Licensed under the GNU GPLv2 license * * The text of the license can be found … Read more
Git prefixes the object with “blob “, followed by the length (as a human-readable integer), followed by a NUL character $ echo -e ‘blob 14\0Hello, World!’ | shasum 8ab686eafeb1f44702738c8b0f24f2567c36da6d Source: http://alblue.bandlem.com/2011/08/git-tip-of-week-objects.html
With the SFTP, running over an encrypted SSH session, there’s negligible chance the file contents could get corrupted while transferring. The SSH itself does data integrity verification. So unless the contents gets corrupted, when reading the local file or writing the remote file, you can be pretty sure that the file was uploaded correctly, if … Read more
… can’t really grasp how the checksum can be used as an identifier for the start of the package (wouldn’t the checksum change all the time?). Yes, the checksum would change since it is derived from the data. But even a fixed-value start-of-packet nibble would (by itself) not be sufficient to (initially) identify (or verify) … Read more
The slowest possible method is to compare two files byte by byte. The fastest I’ve been able to come up with is a similar comparison, but instead of one byte at a time, you would use an array of bytes sized to Int64, and then compare the resulting numbers. Here’s what I came up with: … Read more
You can use hashlib.md5() Note that sometimes you won’t be able to fit the whole file in memory. In that case, you’ll have to read chunks of 4096 bytes sequentially and feed them to the md5 method: import hashlib def md5(fname): hash_md5 = hashlib.md5() with open(fname, “rb”) as f: for chunk in iter(lambda: f.read(4096), b””): … Read more
There’s an input stream decorator, java.security.DigestInputStream, so that you can compute the digest while using the input stream as you normally would, instead of having to make an extra pass over the data. MessageDigest md = MessageDigest.getInstance(“MD5”); try (InputStream is = Files.newInputStream(Paths.get(“file.txt”)); DigestInputStream dis = new DigestInputStream(is, md)) { /* Read decorated stream (dis) to … Read more