How can I use boto to stream a file out of Amazon S3 to Rackspace Cloudfiles?

by

Other answers in this thread are related to boto, but S3.Object is not iterable anymore in boto3. So, the following DOES NOT WORK, it produces an TypeError: 's3.Object' object is not iterable error message:

s3 = boto3.session.Session(profile_name=my_profile).resource('s3')
s3_obj = s3.Object(bucket_name=my_bucket, key=my_key)

with io.FileIO('sample.txt', 'w') as file:
    for i in s3_obj:
        file.write(i)

In boto3, the contents of the object is available at S3.Object.get()['Body'] which is an iterable since version 1.9.68 but previously wasn’t. Thus the following will work for the latest versions of boto3 but not earlier ones:

body = s3_obj.get()['Body']
with io.FileIO('sample.txt', 'w') as file:
    for i in body:
        file.write(i)

So, an alternative for older boto3 versions is to use the read method, but this loads the WHOLE S3 object in memory which when dealing with large files is not always a possibility:

body = s3_obj.get()['Body']
with io.FileIO('sample.txt', 'w') as file:
    for i in body.read():
        file.write(i)

But the read method allows to pass in the amt parameter specifying the number of bytes we want to read from the underlying stream. This method can be repeatedly called until the whole stream has been read:

body = s3_obj.get()['Body']
with io.FileIO('sample.txt', 'w') as file:
    while file.write(body.read(amt=512)):
        pass

Digging into botocore.response.StreamingBody code one realizes that the underlying stream is also available, so we could iterate as follows:

body = s3_obj.get()['Body']
with io.FileIO('sample.txt', 'w') as file:
    for b in body._raw_stream:
        file.write(b)

While googling I’ve also seen some links that could be use, but I haven’t tried:

More Related Contents:

  1. How to write a file or data to an S3 object using boto3
  2. How to upload a file to directory in S3 bucket using boto
  3. Amazon S3 boto – how to delete folder?
  4. Listing contents of a bucket with boto3
  5. Read a file line by line from S3 using boto?
  6. Move files between two AWS S3 buckets using boto3
  7. Can I use boto3 anonymously?
  8. Getting S3 objects’ last modified datetimes with boto
  9. Can you upload to S3 using a stream rather than a local file?
  10. Open S3 object as a string with Boto3
  11. Boto3 to download all files from a S3 Bucket
  12. How to upload a file to Google Cloud Storage on Python 3?
  13. setting up s3 for logs in airflow
  14. How to handle errors with boto3?
  15. Load S3 Data into AWS SageMaker Notebook
  16. How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?
  17. How to save S3 object to a file using boto3
  18. Read file content from S3 bucket with boto3
  19. check if a key exists in a bucket in s3 using boto3
  20. Error “Read-only file system” in AWS Lambda when downloading a file from S3
  21. Retrieving subfolders names in S3 bucket from boto3
  22. AWS Content Type Settings in S3 Using Boto3
  23. How to upload File in FastAPI, then to Amazon S3 and finally process it?
  24. Pandas in AWS lambda gives numpy error
  25. Mocking boto3 S3 client method Python
  26. how to copy s3 object from one bucket to another using python boto3
  27. How to return a PDF file from in-memory buffer using FastAPI?
  28. Python: How to read and load an excel file from AWS S3?
  29. Download a folder from S3 using Boto3
  30. Recommended way to manage credentials with multiple AWS accounts?

Leave a Comment