Best way to join two large datasets in Pandas

by

This seems like a task that dask was designed for. Essentially, dask can do pandas operations out-of-core, so you can work with datasets that don’t fit into memory. The dask.dataframe API is a subset of the pandas API, so there shouldn’t be much of a learning curve. See the Dask DataFrame Overview page for some additional DataFrame specific details.

import dask.dataframe as dd

# Read in the csv files.
df1 = dd.read_csv('file1.csv')
df2 = dd.read_csv('file2.csv')

# Merge the csv files.
df = dd.merge(df1, df2, how='outer', on=['product','version'])

# Write the output.
df.to_csv('file3.csv', index=False)

Assuming that 'product' and 'version' are the only columns, it may be more efficient to replace the merge with:

df = dd.concat([df1, df2]).drop_duplicates()

I’m not entirely sure if that will be better, but apparently merges that aren’t done on the index are “slow-ish” in dask, so it could be worth a try.

More Related Contents:

  1. How to delete multiple pandas (python) dataframes from memory to save RAM?
  2. error when applying function to a series object
  3. How to group dataframe rows into list in pandas groupby
  4. Groupby value counts on the dataframe pandas
  5. Replace values in a pandas series via dictionary efficiently
  6. Pandas: Chained assignments [duplicate]
  7. Apply pandas function to column to create multiple new columns?
  8. Reading an Excel file in python using pandas
  9. Python Pandas: Get index of rows which column matches certain value
  10. How to give a pandas/matplotlib bar graph custom colors
  11. Import multiple excel files into python pandas and concatenate them into one dataframe
  12. Pandas deleting row with df.drop doesn’t work
  13. Random row selection in Pandas dataframe
  14. converting currency with $ to numbers in Python pandas
  15. Lookup Values by Corresponding Column Header in Pandas 1.2.0 or newer
  16. How to spread a column in a Pandas data frame
  17. Python Pandas: Boolean indexing on multiple columns [duplicate]
  18. Vectorized lookup on a pandas dataframe
  19. Find symmetric pairs quickly in numpy
  20. Python pandas convert datetime to timestamp effectively through dt accessor
  21. Checking if particular value (in cell) is NaN in pandas DataFrame not working using ix or iloc
  22. Time Series Analysis – unevenly spaced measures – pandas + statsmodels
  23. Move column by name to front of table in pandas
  24. Convert List to Pandas Dataframe Column
  25. matplotlib savefig() size control
  26. Is there a way to speed up handling large CSVs and dataframes in python?
  27. How to iterate over columns of pandas dataframe to run regression
  28. Load CSV to Pandas MultiIndex DataFrame
  29. Python pandas – new column’s value if the item is in the list
  30. bash: Python import – Command not found for pandas

Leave a Comment