Read a file line by line from S3 using boto?

by

Here’s a solution which actually streams the data line by line:

from io import TextIOWrapper
from gzip import GzipFile
...

# get StreamingBody from botocore.response
response = s3.get_object(Bucket=bucket, Key=key)
# if gzipped
gzipped = GzipFile(None, 'rb', fileobj=response['Body'])
data = TextIOWrapper(gzipped)

for line in data:
    # process line

More Related Contents:

  1. How to write a file or data to an S3 object using boto3
  2. How to upload a file to directory in S3 bucket using boto
  3. Amazon S3 boto – how to delete folder?
  4. Listing contents of a bucket with boto3
  5. Can I use boto3 anonymously?
  6. Getting S3 objects’ last modified datetimes with boto
  7. Open S3 object as a string with Boto3
  8. Boto3 to download all files from a S3 Bucket
  9. How to handle errors with boto3?
  10. How can I use boto to stream a file out of Amazon S3 to Rackspace Cloudfiles?
  11. Load S3 Data into AWS SageMaker Notebook
  12. How to save S3 object to a file using boto3
  13. Read file content from S3 bucket with boto3
  14. Getting started with secure AWS CloudFront streaming with Python
  15. Retrieving subfolders names in S3 bucket from boto3
  16. AWS Content Type Settings in S3 Using Boto3
  17. Move files between two AWS S3 buckets using boto3
  18. How to specify credentials when connecting to boto3 S3?
  19. Can you upload to S3 using a stream rather than a local file?
  20. how to copy s3 object from one bucket to another using python boto3
  21. Reading an JSON file from S3 using Python boto3
  22. Python: How to read and load an excel file from AWS S3?
  23. Download a folder from S3 using Boto3
  24. Recommended way to manage credentials with multiple AWS accounts?
  25. How to import a text file on AWS S3 into pandas without writing to disk
  26. Using psycopg2 with Lambda to Update Redshift (Python)
  27. Save Dataframe to csv directly to s3 Python
  28. Flask, SQLAlchemy : KeyError: ‘SQLALCHEMY_TRACK_MODIFICATIONS’
  29. AWS error from Python: No module named lambda_function
  30. Run manage.py from AWS EB Linux instance

Leave a Comment