Amazon S3 boto – how to delete folder?
Here is 2018 (almost 2019) version: s3 = boto3.resource(‘s3’) bucket = s3.Bucket(‘mybucket’) bucket.objects.filter(Prefix=”myprefix/”).delete()
boto
Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials
try specifying keys manually s3 = boto3.resource(‘s3′, aws_access_key_id=ACCESS_ID, aws_secret_access_key= ACCESS_KEY) Make sure you don’t include your ACCESS_ID and ACCESS_KEY in the code directly for security concerns. Consider using environment configs and injecting them in the code as suggested by @Tiger_Mike. For Prod environments consider using rotating access keys: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey
How to save S3 object to a file using boto3
There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this: s3_client = boto3.client(‘s3’) open(‘hello.txt’).write(‘Hello, world!’) # Upload the file to S3 s3_client.upload_file(‘hello.txt’, ‘MyBucket’, ‘hello-remote.txt’) # Download the file from S3 s3_client.download_file(‘MyBucket’, ‘hello-remote.txt’, ‘hello2.txt’) print(open(‘hello2.txt’).read()) … Read more
AWS: Boto3: AssumeRole example which includes role usage
Here’s a code snippet from the official AWS documentation where an s3 resource is created for listing all s3 buckets. boto3 resources or clients for other services can be built in a similar fashion. # create an STS client object that represents a live connection to the # STS service sts_client = boto3.client(‘sts’) # Call … Read more
How can I use boto to stream a file out of Amazon S3 to Rackspace Cloudfiles?
Other answers in this thread are related to boto, but S3.Object is not iterable anymore in boto3. So, the following DOES NOT WORK, it produces an TypeError: ‘s3.Object’ object is not iterable error message: s3 = boto3.session.Session(profile_name=my_profile).resource(‘s3’) s3_obj = s3.Object(bucket_name=my_bucket, key=my_key) with io.FileIO(‘sample.txt’, ‘w’) as file: for i in s3_obj: file.write(i) In boto3, the contents … Read more
How to handle errors with boto3?
Use the response contained within the exception. Here is an example: import boto3 from botocore.exceptions import ClientError try: iam = boto3.client(‘iam’) user = iam.create_user(UserName=”fred”) print(“Created user: %s” % user) except ClientError as e: if e.response[‘Error’][‘Code’] == ‘EntityAlreadyExists’: print(“User already exists”) else: print(“Unexpected error: %s” % e) The response dict in the exception will contain the … Read more
How to upload a file to directory in S3 bucket using boto
NOTE: This answer uses boto. See the other answer that uses boto3, which is newer. Try this… import boto import boto.s3 import sys from boto.s3.key import Key AWS_ACCESS_KEY_ID = ” AWS_SECRET_ACCESS_KEY = ” bucket_name = AWS_ACCESS_KEY_ID.lower() + ‘-dump’ conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) bucket = conn.create_bucket(bucket_name, location=boto.s3.connection.Location.DEFAULT) testfile = “replace this with an actual filename” print … Read more
How to write a file or data to an S3 object using boto3
In boto 3, the ‘Key.set_contents_from_’ methods were replaced by Object.put() Client.put_object() For example: import boto3 some_binary_data = b’Here we have some data’ more_binary_data = b’Here we have some more data’ # Method 1: Object.put() s3 = boto3.resource(‘s3’) object = s3.Object(‘my_bucket_name’, ‘my/key/including/filename.txt’) object.put(Body=some_binary_data) # Method 2: Client.put_object() client = boto3.client(‘s3′) client.put_object(Body=more_binary_data, Bucket=”my_bucket_name”, Key=’my/key/including/anotherfilename.txt’) Alternatively, the binary … Read more
Amazon S3 boto – how to create a folder?
There is no concept of folders or directories in S3. You can create file names like “abc/xys/uvw/123.jpg”, which many S3 access tools like S3Fox show like a directory structure, but it’s actually just a single file in a bucket.