Filtering out duplicated/non-unique rows in data.table

by

For v1.9.8+ (released November 2016)

From ?unique.data.table
By default all columns are being used (which is consistent with ?unique.data.frame)

unique(dt)
   V1 V2
1:  A  B
2:  A  C
3:  A  D
4:  B  A
5:  C  D
6:  E  F
7:  G  G

Or using the by argument in order to get unique combinations of specific columns (like previously keys were used for)

unique(dt, by = "V2")
   V1 V2
1:  A  B
2:  A  C
3:  A  D
4:  B  A
5:  E  F
6:  G  G

Prior v1.9.8

From ?unique.data.table, it is clear that calling unique on a data table only works on the key. This means you have to reset the key to all columns before calling unique.

library(data.table)
dt <- data.table(
  V1=LETTERS[c(1,1,1,1,2,3,3,5,7,1)],
  V2=LETTERS[c(2,3,4,2,1,4,4,6,7,2)]
)

Calling unique with one column as key:

setkey(dt, "V2")
unique(dt)
     V1 V2
[1,]  B  A
[2,]  A  B
[3,]  A  C
[4,]  A  D
[5,]  E  F
[6,]  G  G

More Related Contents:

  1. data.table with two string columns of set elements, extract unique rows with each row unsorted
  2. Overlap join with start and end positions
  3. Why does data.table update names(DT) by reference, even if I assign to another variable?
  4. data.table objects assigned with := from within function not printed
  5. How to get week numbers from dates?
  6. Sample random rows within each group in a data.table
  7. Faster way to read fixed-width files
  8. Melt using patterns when variable names contain string information – avoid coercion to numeric
  9. Error: package or namespace load failed for ggplot2 and for data.table
  10. Find duplicated rows (based on 2 columns) in Data Frame in R
  11. How do keep only unique words within each string in a vector
  12. No RTools compatible with R version 3.5.0 was found
  13. knitr gets tricked by data.table `:=` assignment
  14. Rolling Sum by Another Variable in R
  15. Merging rows with the same ID variable [duplicate]
  16. Add multiple columns to R data.table in one function call?
  17. Non-equi join using data.table: column missing from the output
  18. Use of lapply .SD in data.table R
  19. Cleaning `Inf` values from an R dataframe
  20. can the value.var in dcast be a list or have multiple value variables?
  21. data.table row-wise sum, mean, min, max like dplyr?
  22. grouped operations that result in length not equal to 1 or length of group in dplyr
  23. `dcast` with empty RHS [duplicate]
  24. How to select R data.table rows based on substring match (a la SQL like)
  25. Subset data.table by logical column
  26. Any way to force fread() of data.table not to stop on empty lines?
  27. R data table: update join
  28. Merging data.tables uses more than 10 GB RAM
  29. Split a string column into several dummy variables
  30. R fast single item lookup from list vs data.table vs hash

Leave a Comment