Compare two DataFrames and output their differences side-by-side

by

The first part is similar to Constantine, you can get the boolean of which rows are empty*:

In [21]: ne = (df1 != df2).any(1)

In [22]: ne
Out[22]:
0    False
1     True
2     True
dtype: bool

Then we can see which entries have changed:

In [23]: ne_stacked = (df1 != df2).stack()

In [24]: changed = ne_stacked[ne_stacked]

In [25]: changed.index.names = ['id', 'col']

In [26]: changed
Out[26]:
id  col
1   score         True
2   isEnrolled    True
    Comment       True
dtype: bool

Here the first entry is the index and the second the columns which has been changed.

In [27]: difference_locations = np.where(df1 != df2)

In [28]: changed_from = df1.values[difference_locations]

In [29]: changed_to = df2.values[difference_locations]

In [30]: pd.DataFrame({'from': changed_from, 'to': changed_to}, index=changed.index)
Out[30]:
               from           to
id col
1  score       1.11         1.21
2  isEnrolled  True        False
   Comment     None  On vacation

* Note: it’s important that df1 and df2 share the same index here. To overcome this ambiguity, you can ensure you only look at the shared labels using df1.index & df2.index, but I think I’ll leave that as an exercise.

More Related Contents:

  1. Pandas conditional creation of a series/dataframe column
  2. How to split a dataframe string column into two columns?
  3. Annotate bars with values on Pandas bar plots
  4. Comparing two dataframes and getting the differences [duplicate]
  5. Compare two columns using pandas
  6. replace() method not working on Pandas DataFrame
  7. set difference for pandas
  8. Nested dictionary to multiindex dataframe where dictionary keys are column labels
  9. Python pandas: how to specify data types when reading an Excel file?
  10. python pandas extract year from datetime: df[‘year’] = df[‘date’].year is not working
  11. Pandas filling missing dates and values within group
  12. Use groupby in Pandas to count things in one column in comparison to another
  13. Combining two Series into a DataFrame in pandas
  14. pandas unique values multiple columns
  15. How to find which columns contain any NaN value in Pandas dataframe
  16. Python Pandas Group by date using datetime data
  17. Convert pandas data frame to series
  18. Pandas – dataframe groupby – how to get sum of multiple columns
  19. Pandas DataFrame merge summing column
  20. Mapping ranges of values in pandas dataframe [duplicate]
  21. pandas: combine two columns in a DataFrame
  22. Binning a column with pandas
  23. Convert month int to month name in Pandas
  24. How to get/set a pandas index column title or name?
  25. Python pandas integer YYYYMMDD to datetime
  26. Pandas groupby and aggregation output should include all the original columns (including the ones not aggregated on)
  27. Selecting a range of columns in a dataframe
  28. Replace NaN with empty list in a pandas dataframe
  29. Get weekday/day-of-week for Datetime column of DataFrame
  30. When is it appropriate to use df.value_counts() vs df.groupby(‘…’).count()?

Leave a Comment