How to “select distinct” across multiple data frame columns in pandas?

by

You can use the drop_duplicates method to get the unique rows in a DataFrame:

In [29]: df = pd.DataFrame({'a':[1,2,1,2], 'b':[3,4,3,5]})

In [30]: df
Out[30]:
   a  b
0  1  3
1  2  4
2  1  3
3  2  5

In [32]: df.drop_duplicates()
Out[32]:
   a  b
0  1  3
1  2  4
3  2  5

You can also provide the subset keyword argument if you only want to use certain columns to determine uniqueness. See the docstring.

More Related Contents:

  1. Remove pandas rows with duplicate indices
  2. Remove duplicates from dataframe, based on two columns A,B, keeping row with max value in another column C
  3. Group duplicate column IDs in pandas dataframe
  4. Remove duplicate rows from Pandas dataframe where only some columns have the same value
  5. How to repeat a Pandas DataFrame?
  6. How are iloc and loc different?
  7. Splitting dataframe into multiple dataframes
  8. Efficient way to unnest (explode) multiple list columns in a pandas DataFrame
  9. Unnest (explode) a Pandas Series
  10. How to convert a dataframe to a dictionary
  11. Pandas new column from groupby averages
  12. Pandas: replace substring in string
  13. Finding non-numeric rows in dataframe in pandas?
  14. Calculate time difference between Pandas Dataframe indices
  15. Factorize a column of strings in pandas
  16. How to get the last N rows of a pandas DataFrame?
  17. Writing large Pandas Dataframes to CSV file in chunks
  18. How to merge a Series and DataFrame
  19. Pretty print a pandas dataframe in VS Code
  20. Transpose the data in a column every nth rows in PANDAS
  21. Add column in dataframe from list
  22. Pandas groupby with bin counts
  23. get first and last values in a groupby
  24. how to convert monthly data to quarterly in pandas
  25. How to constuct a column of data frame recursively with pandas-python?
  26. Python pandas groupby aggregate on multiple columns, then pivot
  27. Create adjacency matrix for two columns in pandas dataframe
  28. Good alternative to Pandas .append() method, now that it is being deprecated?
  29. python pandas- apply function with two arguments to columns
  30. Confusion about pandas copy of slice of dataframe warning

Leave a Comment