boto
Mocking boto3 S3 client method Python
Botocore has a client stubber you can use for just this purpose: docs. Here’s an example of putting an error in: import boto3 from botocore.stub import Stubber client = boto3.client(‘s3’) stubber = Stubber(client) stubber.add_client_error(‘upload_part_copy’) stubber.activate() # Will raise a ClientError client.upload_part_copy() Here’s an example of putting a normal response in. Additionally, the stubber can now … Read more
Read a file line by line from S3 using boto?
Here’s a solution which actually streams the data line by line: from io import TextIOWrapper from gzip import GzipFile … # get StreamingBody from botocore.response response = s3.get_object(Bucket=bucket, Key=key) # if gzipped gzipped = GzipFile(None, ‘rb’, fileobj=response[‘Body’]) data = TextIOWrapper(gzipped) for line in data: # process line
Listing contents of a bucket with boto3
One way to see the contents would be: for my_bucket_object in my_bucket.objects.all(): print(my_bucket_object)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
I’m using Amazon IAM for the particular key ID and access key and just bumped into the same 403 Forbidden… Turns out you need to give permissions that target both the bucket root and its subobjects: { “Statement”: [ { “Principal”: { “AWS”: “*” }, “Effect”: “Allow”, “Action”: “s3:*”, “Resource”: [“arn:aws:s3:::bucket-name/*”, “arn:aws:s3:::bucket-name”] } ] }