What are the differences between feather and parquet?

by

  • Parquet format is designed for long-term storage, where Arrow is more intended for short term or ephemeral storage (Arrow may be more suitable for long-term storage after the 1.0.0 release happens, since the binary format will be stable then)

  • Parquet is more expensive to write than Feather as it features more layers of encoding and compression. Feather is unmodified raw columnar Arrow memory. We will probably add simple compression to Feather in the future.

  • Due to dictionary encoding, RLE encoding, and data page compression, Parquet files will often be much smaller than Feather files

  • Parquet is a standard storage format for analytics that’s supported by many different systems: Spark, Hive, Impala, various AWS services, in future by BigQuery, etc. So if you are doing analytics, Parquet is a good option as a reference storage format for query by multiple systems

The benchmarks you showed are going to be very noisy since the data you read and wrote is very small. You should try compressing at least 100MB or upwards 1GB of data to get some more informative benchmarks, see e.g. http://wesmckinney.com/blog/python-parquet-multithreading/

More Related Contents:

  1. How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?
  2. How to read a Parquet file into Pandas DataFrame?
  3. How to read partitioned parquet files from S3 using pyarrow in python
  4. Pandas read_csv dtype read all columns but few as string
  5. Convert pandas DataFrame into list of lists [duplicate]
  6. Can you format pandas integers for display, like `pd.options.display.float_format` for floats?
  7. Got continuous is not supported error in RandomForestRegressor
  8. Pandas groupby and aggregation output should include all the original columns (including the ones not aggregated on)
  9. Python: Datetime to season
  10. Round float columns in pandas dataframe
  11. Converting PANDAS dataframe from monthly to daily
  12. Merging two tables with millions of rows in Python
  13. computing the mean for python datetime
  14. How to convert string representation of dictionary in Pandas DataFrame to a new columns?
  15. Get the row corresponding to the max in pandas GroupBy [duplicate]
  16. Copy text between parentheses in pandas DataFrame column into another column
  17. Selecting a range of columns in a dataframe
  18. countplot() with frequencies
  19. Get frequency of item occurrences in a column as percentage [duplicate]
  20. Can I perform dynamic cumsum of rows in pandas?
  21. How to set the xlim and ylim of a FacetGrid
  22. Problem in combining bar plot and line plot
  23. Pandas explode multiple columns [duplicate]
  24. Horizontal stacked bar chart in Matplotlib
  25. Get weekday/day-of-week for Datetime column of DataFrame
  26. pandas combine two strings ignore nan values
  27. Plot all pandas dataframe columns separately
  28. Python pandas equivalent for replace
  29. Pandas data frame to dictionary of lists
  30. Pandas min() of selected row and columns

Leave a Comment