Boto3 to download all files from a S3 Bucket

by

I have the same needs and created the following function that download recursively the files.

The directories are created locally only if they contain files.

import boto3
import os

def download_dir(client, resource, dist, local="/tmp", bucket="your_bucket"):
    paginator = client.get_paginator('list_objects')
    for result in paginator.paginate(Bucket=bucket, Delimiter="https://stackoverflow.com/", Prefix=dist):
        if result.get('CommonPrefixes') is not None:
            for subdir in result.get('CommonPrefixes'):
                download_dir(client, resource, subdir.get('Prefix'), local, bucket)
        for file in result.get('Contents', []):
            dest_pathname = os.path.join(local, file.get('Key'))
            if not os.path.exists(os.path.dirname(dest_pathname)):
                os.makedirs(os.path.dirname(dest_pathname))
            if not file.get('Key').endswith("https://stackoverflow.com/"):
                resource.meta.client.download_file(bucket, file.get('Key'), dest_pathname)

The function is called that way:

def _start():
    client = boto3.client('s3')
    resource = boto3.resource('s3')
    download_dir(client, resource, 'clientconf/', '/tmp', bucket="my-bucket")

More Related Contents:

  1. How to write a file or data to an S3 object using boto3
  2. Read file content from S3 bucket with boto3
  3. Listing contents of a bucket with boto3
  4. Retrieving subfolders names in S3 bucket from boto3
  5. Can I use boto3 anonymously?
  6. Getting S3 objects’ last modified datetimes with boto
  7. How to specify credentials when connecting to boto3 S3?
  8. Open S3 object as a string with Boto3
  9. how to copy s3 object from one bucket to another using python boto3
  10. Reading an JSON file from S3 using Python boto3
  11. Download a folder from S3 using Boto3
  12. How to import a text file on AWS S3 into pandas without writing to disk
  13. How to upload a file to directory in S3 bucket using boto
  14. Complete scan of dynamoDb with boto3
  15. How to handle errors with boto3?
  16. Save Dataframe to csv directly to s3 Python
  17. Load S3 Data into AWS SageMaker Notebook
  18. How to choose an AWS profile when using boto3 to connect to CloudFront
  19. How to save S3 object to a file using boto3
  20. check if a key exists in a bucket in s3 using boto3
  21. Error “Read-only file system” in AWS Lambda when downloading a file from S3
  22. Getting started with secure AWS CloudFront streaming with Python
  23. Amazon S3 boto – how to delete folder?
  24. AWS Content Type Settings in S3 Using Boto3
  25. Read a file line by line from S3 using boto?
  26. How to upload File in FastAPI, then to Amazon S3 and finally process it?
  27. How to return a PDF file from in-memory buffer using FastAPI?
  28. Python: How to read and load an excel file from AWS S3?
  29. unable to call firefox from selenium in python on AWS machine
  30. How to bootstrap installation of Python modules on Amazon EMR?

Leave a Comment