Pandas: Drop consecutive duplicates

by

Use shift:

a.loc[a.shift(-1) != a]

Out[3]:

1    1
3    2
4    3
5    2
dtype: int64

So the above uses boolean critieria, we compare the dataframe against the dataframe shifted by -1 rows to create the mask

Another method is to use diff:

In [82]:

a.loc[a.diff() != 0]
Out[82]:
1    1
2    2
4    3
5    2
dtype: int64

But this is slower than the original method if you have a large number of rows.

Update

Thanks to Bjarke Ebert for pointing out a subtle error, I should actually use shift(1) or just shift() as the default is a period of 1, this returns the first consecutive value:

In [87]:

a.loc[a.shift() != a]
Out[87]:
1    1
2    2
4    3
5    2
dtype: int64

Note the difference in index values, thanks @BjarkeEbert!

More Related Contents:

  1. Adding value labels on a matplotlib bar chart
  2. Pandas column of lists, create a row for each list element
  3. python pandas remove duplicate columns
  4. How do I get a list of all the duplicate items using pandas in python?
  5. How to take column-slices of dataframe in pandas
  6. Pandas groupby multiple fields then diff
  7. Shuffle DataFrame rows
  8. Creating a Pandas DataFrame from a Numpy array: How do I specify the index column and column headers?
  9. Remove duplicates from dataframe, based on two columns A,B, keeping row with max value in another column C
  10. Concatenate a list of pandas dataframes together
  11. How do I convert strings in a Pandas data frame to a ‘date’ data type?
  12. How to normalize json correctly by Python Pandas
  13. Replace invalid values with None in Pandas DataFrame
  14. How to show all columns’ names on a large pandas dataframe?
  15. Pandas monthly rolling operation
  16. Count number of words per row
  17. How to select all columns except one in pandas?
  18. Shift column in pandas dataframe up by one?
  19. In Pandas, does .iloc method give a copy or view?
  20. Move non-empty cells to the left in pandas DataFrame
  21. pandas logical and operator with and without brackets produces different results [duplicate]
  22. How to json_normalize a column with NaNs
  23. Convert Pandas Series to DateTime in a DataFrame
  24. How to use a conditional statement based on DataFrame boolean value in pandas
  25. Stacked bar chart in Seaborn
  26. How to add/subtract time (hours, minutes, etc.) from a Pandas DataFrame.Index whos objects are of type datetime.time?
  27. Remove substring from column based on another column
  28. plotting value_counts() in seaborn barplot
  29. How to resample a dataframe with different functions applied to each column?
  30. Convert a column of timestamps into periods in pandas

Leave a Comment