How to import a text file on AWS S3 into pandas without writing to disk

by

pandas uses boto for read_csv, so you should be able to:

import boto
data = pd.read_csv('s3://bucket....csv')

If you need boto3 because you are on python3.4+, you can

import boto3
import io
s3 = boto3.client('s3')
obj = s3.get_object(Bucket="bucket", Key='key')
df = pd.read_csv(io.BytesIO(obj['Body'].read()))

Since version 0.20.1 pandas uses s3fs, see answer below.

More Related Contents:

  1. How to write a file or data to an S3 object using boto3
  2. Boto3 to download all files from a S3 Bucket
  3. Save Dataframe to csv directly to s3 Python
  4. Read file content from S3 bucket with boto3
  5. check if a key exists in a bucket in s3 using boto3
  6. Error “Read-only file system” in AWS Lambda when downloading a file from S3
  7. Listing contents of a bucket with boto3
  8. Retrieving subfolders names in S3 bucket from boto3
  9. How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?
  10. How to upload File in FastAPI, then to Amazon S3 and finally process it?
  11. Can I use boto3 anonymously?
  12. Pandas in AWS lambda gives numpy error
  13. Getting S3 objects’ last modified datetimes with boto
  14. How to specify credentials when connecting to boto3 S3?
  15. Open S3 object as a string with Boto3
  16. how to copy s3 object from one bucket to another using python boto3
  17. How to return a PDF file from in-memory buffer using FastAPI?
  18. Reading an JSON file from S3 using Python boto3
  19. Download a folder from S3 using Boto3
  20. how to calculation cost time [closed]
  21. Reshape wide to long in pandas
  22. How do convert a pandas dataframe to XML?
  23. How to add hovering annotations to a plot
  24. Converting between datetime and Pandas Timestamp objects
  25. Shift NaNs to the end of their respective rows
  26. multi index plotting
  27. Pandas how can ‘replace’ work after ‘loc’?
  28. Pandas Dataframe: split column into multiple columns, right-align inconsistent cell entries
  29. Pandas: Reading Excel with merged cells
  30. Creating multiple Excel worksheets using data from a pandas DataFrame

Leave a Comment