How to save S3 object to a file using boto3

by

There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this:

s3_client = boto3.client('s3')
open('hello.txt').write('Hello, world!')

# Upload the file to S3
s3_client.upload_file('hello.txt', 'MyBucket', 'hello-remote.txt')

# Download the file from S3
s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')
print(open('hello2.txt').read())

These functions will automatically handle reading/writing files as well as doing multipart uploads in parallel for large files.

Note that s3_client.download_file won’t create a directory. It can be created as pathlib.Path('/path/to/file.txt').parent.mkdir(parents=True, exist_ok=True).

More Related Contents:

  1. How to write a file or data to an S3 object using boto3
  2. How to handle errors with boto3?
  3. Listing contents of a bucket with boto3
  4. Can I use boto3 anonymously?
  5. Getting S3 objects’ last modified datetimes with boto
  6. Open S3 object as a string with Boto3
  7. Boto3 to download all files from a S3 Bucket
  8. How to upload a file to directory in S3 bucket using boto
  9. Complete scan of dynamoDb with boto3
  10. How to choose an AWS profile when using boto3 to connect to CloudFront
  11. Read file content from S3 bucket with boto3
  12. Amazon S3 boto – how to delete folder?
  13. Retrieving subfolders names in S3 bucket from boto3
  14. Read a file line by line from S3 using boto?
  15. How to specify credentials when connecting to boto3 S3?
  16. Mocking boto3 S3 client method Python
  17. how to copy s3 object from one bucket to another using python boto3
  18. Reading an JSON file from S3 using Python boto3
  19. Download a folder from S3 using Boto3
  20. Recommended way to manage credentials with multiple AWS accounts?
  21. unable to call firefox from selenium in python on AWS machine
  22. Error “Read-only file system” in AWS Lambda when downloading a file from S3
  23. AWS Content Type Settings in S3 Using Boto3
  24. How to upload File in FastAPI, then to Amazon S3 and finally process it?
  25. Move files between two AWS S3 buckets using boto3
  26. How to bootstrap installation of Python modules on Amazon EMR?
  27. Flask, SQLAlchemy : KeyError: ‘SQLALCHEMY_TRACK_MODIFICATIONS’
  28. Python: How to read and load an excel file from AWS S3?
  29. AWS error from Python: No module named lambda_function
  30. Run manage.py from AWS EB Linux instance

Leave a Comment