What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?

by

Say you uploaded a 14MB file to a bucket without server-side encryption, and your part size is 5MB. Calculate 3 MD5 checksums corresponding to each part, i.e. the checksum of the first 5MB, the second 5MB, and the last 4MB. Then take the checksum of their concatenation. MD5 checksums are often printed as hex representations of binary data, so make sure you take the MD5 of the decoded binary concatenation, not of the ASCII or UTF-8 encoded concatenation. When that’s done, add a hyphen and the number of parts to get the ETag.

Here are the commands to do it on Mac OS X from the console:

$ dd bs=1m count=5 skip=0 if=someFile | md5 >>checksums.txt
5+0 records in
5+0 records out
5242880 bytes transferred in 0.019611 secs (267345449 bytes/sec)
$ dd bs=1m count=5 skip=5 if=someFile | md5 >>checksums.txt
5+0 records in
5+0 records out
5242880 bytes transferred in 0.019182 secs (273323380 bytes/sec)
$ dd bs=1m count=5 skip=10 if=someFile | md5 >>checksums.txt
2+1 records in
2+1 records out
2599812 bytes transferred in 0.011112 secs (233964895 bytes/sec)

At this point all the checksums are in checksums.txt. To concatenate them and decode the hex and get the MD5 checksum of the lot, just use

$ xxd -r -p checksums.txt | md5

And now append “-3” to get the ETag, since there were 3 parts.

Notes

  • If you uploaded with aws-cli via aws s3 cp then you most likely have a 8MB chunksize. According to the docs, that is the default.
  • If the bucket has server-side encryption (SSE) turned on, the ETag won’t be the MD5 checksum (see the API documentation). But if you’re just trying to verify that an uploaded part matches what you sent, you can use the Content-MD5 header and S3 will compare it for you.
  • md5 on macOS just writes out the checksum, but md5sum on Linux/brew also outputs the filename. You’ll need to strip that, but I’m sure there’s some option to only output the checksums. You don’t need to worry about whitespace cause xxd will ignore it.

Code Links

More Related Contents:

  1. Amazon S3 boto – how to create a folder?
  2. Set cache-control for entire S3 bucket automatically (using bucket policies?)
  3. Correct S3 + Cloudfront CORS Configuration?
  4. Exporting data from Google Cloud Storage to Amazon S3
  5. Quick way to list all files in Amazon S3 bucket?
  6. how to add cache control in AWS S3?
  7. How to get the md5sum of a file on Amazon’s S3
  8. Receive AccessDenied when trying to access a reload or refresh or one in new tab in angular 5
  9. Is there an S3 policy for limiting access to only see/access one bucket?
  10. How do I have an S3 bucket return 404 (instead of 403) for a key that does not exist in the bucket/
  11. AWS S3 copy files and folders between two buckets
  12. How to call Amazon S3 bucket using postdata preSigned Url to upload a file using Karate
  13. Getting Access Denied when calling the PutObject operation with bucket-level permission
  14. S3 Error: The difference between the request time and the current time is too large
  15. How do I delete a versioned bucket in AWS S3 using the CLI?
  16. Best way to move files between S3 buckets?
  17. Amazon S3: Static Web Sites: Custom Domain or Subdomain
  18. What are valid S3 key names that can be accessed via the S3 rest API?
  19. Why are S3 and Google Storage bucket names a global namespace?
  20. Restrict List of Buckets for a Specific User
  21. Directory Listing in S3 Static Website
  22. CloudFront + S3 Website: “The specified key does not exist” when an implicit index document should be displayed
  23. Javascript to download a file from amazon s3 bucket?
  24. Load S3 Data into AWS SageMaker Notebook
  25. Append data to an S3 object
  26. Error “Read-only file system” in AWS Lambda when downloading a file from S3
  27. Serving a multitude of static sites from a wildcard domain in AWS
  28. Pandas in AWS lambda gives numpy error
  29. AWS Glue output file name
  30. upload file from angularjs directly to amazon s3 using signed url

Leave a Comment