Save Dataframe to csv directly to s3 Python

by

You can use:

from io import StringIO # python3; python2: BytesIO 
import boto3

bucket="my_bucket_name" # already created on S3
csv_buffer = StringIO()
df.to_csv(csv_buffer)
s3_resource = boto3.resource('s3')
s3_resource.Object(bucket, 'df.csv').put(Body=csv_buffer.getvalue())

More Related Contents:

  1. Import multiple csv files into pandas and concatenate into one DataFrame
  2. How to add pandas data to an existing csv file?
  3. Writing a pandas DataFrame to CSV file
  4. Import CSV file as a pandas DataFrame
  5. How to make separator in pandas read_csv more flexible wrt whitespace, for irregular separators?
  6. How to get rid of “Unnamed: 0” column in a pandas DataFrame read in from CSV file?
  7. What is the fastest way to upload a big csv file in notebook to work with python pandas?
  8. Pandas split column into multiple columns by comma
  9. Boto3 to download all files from a S3 Bucket
  10. How to import a text file on AWS S3 into pandas without writing to disk
  11. Read file content from S3 bucket with boto3
  12. pandas DataFrame output end of csv
  13. Loading multiple csv files of a folder into one dataframe
  14. Listing contents of a bucket with boto3
  15. Retrieving subfolders names in S3 bucket from boto3
  16. How to upload File in FastAPI, then to Amazon S3 and finally process it?
  17. Can I use boto3 anonymously?
  18. How to add header row to a pandas DataFrame
  19. How to specify credentials when connecting to boto3 S3?
  20. Open S3 object as a string with Boto3
  21. How to return a PDF file from in-memory buffer using FastAPI?
  22. Is there a way to speed up handling large CSVs and dataframes in python?
  23. How do you remove the column name row when exporting a pandas DataFrame?
  24. Reading an JSON file from S3 using Python boto3
  25. Calculate summary statistics of columns in dataframe
  26. pandas dataframe with 2-rows header and export to csv
  27. UnicodeDecodeError when reading CSV file in Pandas
  28. How to return a csv file/Pandas DataFrame in JSON format using FastAPI?
  29. start index at 1 for Pandas DataFrame
  30. Download a folder from S3 using Boto3

Leave a Comment