Why is rbindlist “better” than rbind?

by

rbindlist is an optimized version of do.call(rbind, list(...)), which is known for being slow when using rbind.data.frame

Where does it really excel

Some questions that show where rbindlist shines are

Fast vectorized merge of list of data.frames by row

Trouble converting long list of data.frames (~1 million) to single data.frame using do.call and ldply

These have benchmarks that show how fast it can be.

rbind.data.frame is slow, for a reason

rbind.data.frame does lots of checking, and will match by name. (i.e. rbind.data.frame will account for the fact that columns may be in different orders, and match up by name), rbindlist doesn’t do this kind of checking, and will join by position

eg

do.call(rbind, list(data.frame(a = 1:2, b = 2:3), data.frame(b = 1:2, a = 2:3)))
##    a b
## 1  1 2
## 2  2 3
## 3  2 1
## 4  3 2

rbindlist(list(data.frame(a = 1:5, b = 2:6), data.frame(b = 1:5, a = 2:6)))
##     a b
##  1: 1 2
##  2: 2 3
##  3: 1 2
##  4: 2 3

Some other limitations of rbindlist

It used to struggle to deal with factors, due to a bug that has since been fixed:

rbindlist two data.tables where one has factor and other has character type for a column (Bug #2650)

It has problems with duplicate column names

see
Warning message: in rbindlist(allargs) : NAs introduced by coercion: possible bug in data.table? (Bug #2384)

rbind.data.frame rownames can be frustrating

rbindlist can handle lists data.frames and data.tables, and will return a data.table without rownames

you can get in a muddle of rownames using do.call(rbind, list(...))
see

How to avoid renaming of rows when using rbind inside do.call?

Memory efficiency

In terms of memory rbindlist is implemented in C, so is memory efficient, it uses setattr to set attributes by reference

rbind.data.frame is implemented in R, it does lots of assigning, and uses attr<- (and class<- and rownames<- all of which will (internally) create copies of the created data.frame.

More Related Contents:

  1. Efficient way to rbind data.frames with different columns
  2. Aggregate / summarize multiple variables per group (e.g. sum, mean)
  3. Replacing NAs with latest non-NA value
  4. Getting the top values by group
  5. Using data.table package inside my own package
  6. Faster version of combn
  7. Pass column name in data.table using variable [duplicate]
  8. What does .SD stand for in data.table in R
  9. What’s wrong with my function to load multiple .csv files into single dataframe in R using rbind?
  10. Apply multiple functions to multiple columns in data.table
  11. Converting nested list to dataframe
  12. data.table – select first n rows within group [duplicate]
  13. subsetting a data.table using != excludes NA too
  14. Dynamic column names in data.table
  15. Merge rows in a dataframe where the rows are disjoint and contain NAs
  16. Calculate row-wise maximum
  17. Convert *some* column classes in data.table
  18. Difference between rbind() and bind_rows() in R
  19. Canonical tidyverse method to update some values of a vector from a look-up table
  20. write.csv for large data.table
  21. Insert a row in a data.table
  22. Subsetting data.table set by date range in R
  23. Extracting unique rows from a data table in R [duplicate]
  24. Accessing columns in data.table using a character vector of column names
  25. How can I evaluate (or create) an on the fly column in data.table in r
  26. Keyed lookup on data.table without ‘with’
  27. Subset data.table by logical column
  28. Merging data.tables uses more than 10 GB RAM
  29. Is there a _fast_ way to run a rolling regression inside data.table?
  30. Fill option for fread

Leave a Comment