Reading 40 GB csv file into R using bigmemory

by

I don’t know about bigmemory, but to satisfy your challenges you don’t need to read the file in. Simply pipe some bash/awk/sed/python/whatever processing to do the steps you want, i.e. throw out NULL lines and randomly select N lines, and then read that in.

Here’s an example using awk (assuming you want 100 random lines from a file that has 1M lines).

read.csv(pipe('awk -F, \'BEGIN{srand(); m = 100; length = 1000000;}
                       !/NULL/{if (rand() < m/(length - NR + 1)) {
                                 print; m--;
                                 if (m == 0) exit;
                              }}\' filename'
        )) -> df

It wasn’t obvious to me what you meant by NULL, so I used literal understanding of it, but it should be easy to modify it to fit your needs.

More Related Contents:

  1. R memory management / cannot allocate vector of size n Mb
  2. Increasing (or decreasing) the memory available to R processes
  3. How to escape backslashes in R string
  4. Read a text file in R line by line
  5. Memory Allocation “Error: cannot allocate vector of size 75.1 Mb” [duplicate]
  6. Escaping backslash (\) in string or paths in R
  7. How to source() .R file saved using UTF-8 encoding?
  8. What is the difference between gc() and rm()
  9. Reading two-line headers in R
  10. Moving files between folders
  11. write.csv for large data.table
  12. R: how to rbind two huge data-frames without running out of memory
  13. Override column types when importing data using readr::read_csv() when there are many columns
  14. Save multiple ggplots using a for loop
  15. Plotting contours on an irregular grid
  16. How to merge color, line style and shape legends in ggplot
  17. Horizontal/Vertical Line in plotly
  18. Compute mean and standard deviation by group for multiple variables in a data.frame
  19. Fill and border colour in geom_point (scale_colour_manual) in ggplot
  20. R ggplot2 merge with shapefile and csv data to fill polygons
  21. lambda-like functions in R?
  22. remove all line breaks (enter symbols) from the string using R
  23. Error in loadNamespace(name) : there is no package called ‘Rcpp’
  24. RStudio not picking the encoding I’m telling it to use when reading a file
  25. Is there a dictionary functionality in R
  26. Converting unit abbreviations to numbers
  27. How to update a package in R?
  28. R: eval(parse(…)) is often suboptimal
  29. Sum object in a column between an interval defined by another column
  30. R shiny sliderInput with restricted range

Leave a Comment