Set cache-control for entire S3 bucket automatically (using bucket policies?)

by

There are now 3 ways to get this done: via the AWS Console, via the command line, or via the s3cmd command line tool.

AWS Console Instructions

This is now the recommended solution. It is straight forward, but it can take some time.

  • Log in to AWS Management Console
  • Go into S3 bucket
  • Select all files by route
  • Choose “More” from the menu
  • Select “Change metadata”
  • In the “Key” field, select “Cache-Control” from the drop down menu
    max-age=604800 Enter (7 days) for Value
  • Press “Save” button

(thanks to @biplob – please give him some love below)

AWS Command Line Solution

Originally, when I created this bucket policies were a no go, so I figured how to do it using aws-cli, and it is pretty slick. When researching I couldn’t find any examples in the wild, so I thought I would post some of my solutions to help those in need.

NOTE: By default, aws-cli only copies a file’s current metadata, EVEN IF YOU SPECIFY NEW METADATA.

To use the metadata that is specified on the command line, you need to add the ‘–metadata-directive REPLACE’ flag. Here are a some examples.

For a single file

aws s3 cp s3://mybucket/file.txt s3://mybucket/file.txt --metadata-directive REPLACE \
--expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

For an entire bucket (note –recursive flag):

aws s3 cp s3://mybucket/ s3://mybucket/ --recursive --metadata-directive REPLACE \
--expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

A little gotcha I found, if you only want to apply it to a specific file type, you need to exclude all the files, then include the ones you want.

Only jpgs and pngs:

aws s3 cp s3://mybucket/ s3://mybucket/ --exclude "*" --include "*.jpg" --include "*.png" \
--recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read \
--cache-control max-age=2592000,public

Here are some links to the manual if you need more info:

Known Issues:

"Unknown options: --metadata-directive, REPLACE"

this can be caused by an out of date awscli – see @eliotRosewater’s answer below

S3cmd tool

S3cmd is a “Command line tool for managing Amazon S3 and CloudFront services”. While this solution requires a git pull it might be a simpler and more comprehensive solution.

For full instructions, see @ashishyadaveee11’s post below

Hope it helps!

More Related Contents:

  1. What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?
  2. Amazon S3 boto – how to create a folder?
  3. Correct S3 + Cloudfront CORS Configuration?
  4. Exporting data from Google Cloud Storage to Amazon S3
  5. Quick way to list all files in Amazon S3 bucket?
  6. How to get the md5sum of a file on Amazon’s S3
  7. Receive AccessDenied when trying to access a reload or refresh or one in new tab in angular 5
  8. Is there an S3 policy for limiting access to only see/access one bucket?
  9. How do I have an S3 bucket return 404 (instead of 403) for a key that does not exist in the bucket/
  10. AWS S3 copy files and folders between two buckets
  11. How to call Amazon S3 bucket using postdata preSigned Url to upload a file using Karate
  12. Getting Access Denied when calling the PutObject operation with bucket-level permission
  13. S3 Error: The difference between the request time and the current time is too large
  14. Best way to move files between S3 buckets?
  15. Amazon S3: Static Web Sites: Custom Domain or Subdomain
  16. How stable is s3fs to mount an Amazon S3 bucket as a local directory [closed]
  17. What are valid S3 key names that can be accessed via the S3 rest API?
  18. Why are S3 and Google Storage bucket names a global namespace?
  19. Restrict List of Buckets for a Specific User
  20. Directory Listing in S3 Static Website
  21. Nginx proxy Amazon S3 resources
  22. Downloading an entire S3 bucket?
  23. How can I tell how many objects I’ve stored in an S3 bucket?
  24. AWS Content Type Settings in S3 Using Boto3
  25. S3 – What Exactly Is A Prefix? And what Ratelimits apply?
  26. AWS S3 Bucket Access from EC2
  27. Converting mysql table to spark dataset is very slow compared to same from csv file
  28. Amazon AWS Filezilla transfer permission denied
  29. How do I find the total size of my AWS S3 storage bucket or folder?
  30. Retrieve S3 file as Object instead of downloading to absolute system path

Leave a Comment