Practical limits of R data frame

by

R is suited for large data sets, but you may have to change your way of working somewhat from what the introductory textbooks teach you. I did a post on Big Data for R which crunches a 30 GB data set and which you may find useful for inspiration.

The usual sources for information to get started are High-Performance Computing Task View and the R-SIG HPC mailing list at R-SIG HPC.

The main limit you have to work around is a historic limit on the length of a vector to 2^31-1 elements which wouldn’t be so bad if R did not store matrices as vectors. (The limit is for compatibility with some BLAS libraries.)

We regularly analyse telco call data records and marketing databases with multi-million customers using R, so would be happy to talk more if you are interested.

More Related Contents:

  1. Split a large dataframe into a list of data frames based on common value in column
  2. Fastest way to replace NAs in a large data.table
  3. How can I prevent rbind() from geting really slow as dataframe grows larger?
  4. The difference between bracket [ ] and double bracket [[ ]] for accessing the elements of a list or dataframe
  5. Numbering rows within groups in a data frame
  6. Calculate the mean by group
  7. Convert data from long format to wide format with multiple measure columns
  8. Convert row names into first column
  9. Add row to a data frame with total sum for each column
  10. Subset dataframe by multiple logical conditions of rows to remove
  11. Rename multiple columns by names
  12. Merge two data frames while keeping the original row order
  13. Call apply-like function on each row of dataframe with multiple arguments from each row
  14. Why am I getting X. in my column names when reading a data frame?
  15. How to subset data in R without losing NA rows?
  16. How to delete columns that contain ONLY NAs?
  17. Using grep to help subset a data frame
  18. RcppArmadillo pass user-defined function
  19. Add an index (numeric ID) column to large data frame [duplicate]
  20. How to add multiple columns to a data.frame in one go?
  21. Splitting a string into new rows in R [duplicate]
  22. Transforming a time-series into a data frame and back
  23. Split a column into multiple binary dummy columns [duplicate]
  24. Split up a dataframe by number of rows
  25. Merge data frames based on rownames in R
  26. dplyr change many data types
  27. Adding new column with conditional values using ifelse
  28. Return pmin or pmax of data.frame with multiple columns
  29. How to divide a number of columns by one column?
  30. Split a column by group [duplicate]

Leave a Comment