How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?

by

You should use the s3fs module as proposed by yjk21. However as result of calling ParquetDataset you’ll get a pyarrow.parquet.ParquetDataset object. To get the Pandas DataFrame you’ll rather want to apply .read_pandas().to_pandas() to it:

import pyarrow.parquet as pq
import s3fs
s3 = s3fs.S3FileSystem()

pandas_dataframe = pq.ParquetDataset('s3://your-bucket/', filesystem=s3).read_pandas().to_pandas()

More Related Contents:

  1. Why does the following code cant get value of n_fold = 1?
  2. How to change the order of DataFrame columns?
  3. Filter dataframe rows if value in column is in a set list of values [duplicate]
  4. Add missing dates to pandas dataframe
  5. Converting a Pandas GroupBy output from Series to DataFrame
  6. Pandas: sum DataFrame rows for given columns
  7. String concatenation of two pandas columns
  8. datetime dtypes in pandas read_csv
  9. Pandas DataFrame: replace all values in a column, based on condition
  10. Pandas – Slice large dataframe into chunks
  11. Calculate average of every x rows in a table and create new table
  12. Find column name in pandas that matches an array
  13. Removing prefix from column names in Pandas
  14. pandas get the row-wise minimum value of two or more columns
  15. Set maximum value (upper bound) in pandas DataFrame
  16. Normalizing pandas DataFrame rows by their sums
  17. How to merge multiple dataframes
  18. How can I get a value from a cell of a dataframe?
  19. Return multiple columns from pandas apply()
  20. How to slice a pandas DataFrame by position?
  21. Python Pandas – Merge based on substring in string
  22. Filling in date gaps in MultiIndex Pandas Dataframe
  23. Writing a Python Pandas DataFrame to Word document
  24. Find unique values in a Pandas dataframe, irrespective of row or column location
  25. Pandas groupby and aggregation output should include all the original columns (including the ones not aggregated on)
  26. Selecting a range of columns in a dataframe
  27. Replace NaN with empty list in a pandas dataframe
  28. Get weekday/day-of-week for Datetime column of DataFrame
  29. Get max value from row of a dataframe in python [duplicate]
  30. When is it appropriate to use df.value_counts() vs df.groupby(‘…’).count()?

Leave a Comment