How can I prevent rbind() from geting really slow as dataframe grows larger?

by

You are in the 2nd circle of hell, namely failing to pre-allocate data structures.

Growing objects in this fashion is a Very Very Bad Thing in R. Either pre-allocate and insert:

df <- data.frame(x = rep(NA,20000),y = rep(NA,20000))

or restructure your code to avoid this sort of incremental addition of rows. As discussed at the link I cite, the reason for the slowness is that each time you add a row, R needs to find a new contiguous block of memory to fit the data frame in. Lots ‘o copying.

More Related Contents:

  1. Split a large dataframe into a list of data frames based on common value in column
  2. Fastest way to replace NAs in a large data.table
  3. rbind dataframes with a different column name
  4. Append an object to a list in R in amortized constant time, O(1)?
  5. Converting nested list to dataframe
  6. How to append rows to an R data frame
  7. Practical limits of R data frame
  8. rbind data frames based on a common pattern in data frame name
  9. Combine several data frames in the global environment by row (rbind)
  10. Merging a lot of data.frames [duplicate]
  11. dplyr: nonstandard column names (white space, punctuation, starts with numbers)
  12. Remove columns with zero values from a dataframe
  13. Calculating statistics on subsets of data [duplicate]
  14. How to convert dataframe into time series?
  15. Identifying duplicate columns in a dataframe
  16. Merge unequal dataframes and replace missing rows with 0
  17. R pass variable column indices to ggplot2 [duplicate]
  18. How to access the last value in a vector?
  19. Removing display of row names from data frame
  20. Why is plyr so slow?
  21. How to delete rows from a data.frame, based on an external list, using R?
  22. How to add multiple columns to a data.frame in one go?
  23. Determine the data types of a data frame’s columns
  24. Changing Million/Billion abbreviations into actual numbers? ie. 5.12M -> 5,120,000 [duplicate]
  25. Extract text after “/” in a data frame column
  26. Merge overlapping ranges into unique groups, in dataframe
  27. Deleting specific rows from a data frame
  28. Count number of non-NA values for every column in a dataframe [duplicate]
  29. Efficient string similarity grouping
  30. What you can do with a data.frame that you can’t with a data.table?

Leave a Comment