lapply vs for loop – Performance R

by

First of all, it is an already long debunked myth that for loops are any slower than lapply. The for loops in R have been made a lot more performant and are currently at least as fast as lapply.

That said, you have to rethink your use of lapply here. Your implementation demands assigning to the global environment, because your code requires you to update the weight during the loop. And that is a valid reason to not consider lapply.

lapply is a function you should use for its side effects (or lack of side effects). The function lapply combines the results in a list automatically and doesn’t mess with the environment you work in, contrary to a for loop. The same goes for replicate. See also this question:

Is R’s apply family more than syntactic sugar?

The reason your lapply solution is far slower, is because your way of using it creates a lot more overhead.

  • replicate is nothing else but sapply internally, so you actually combine sapply and lapply to implement your double loop. sapply creates extra overhead because it has to test whether or not the result can be simplified. So a for loop will be actually faster than using replicate.
  • inside your lapply anonymous function, you have to access the dataframe for both x and y for every observation. This means that -contrary to in your for-loop- eg the function $ has to be called every time.
  • Because you use these high-end functions, your ‘lapply’ solution calls 49 functions, compared to your for solution that only calls 26. These extra functions for the lapply solution include calls to functions like match, structure, [[, names, %in%, sys.call, duplicated, …
    All functions not needed by your for loop as that one doesn’t do any of these checks.

If you want to see where this extra overhead comes from, look at the internal code of replicate, unlist, sapply and simplify2array.

You can use the following code to get a better idea of where you lose your performance with the lapply. Run this line by line!

Rprof(interval = 0.0001)
f()
Rprof(NULL)
fprof <- summaryRprof()$by.self

Rprof(interval = 0.0001)
perceptron(as.matrix(irissubdf[1:2]), irissubdf$y, 1, 10) 
Rprof(NULL)
perprof <- summaryRprof()$by.self

fprof$Fun <- rownames(fprof)
perprof$Fun <- rownames(perprof)

Selftime <- merge(fprof, perprof,
                  all = TRUE,
                  by = 'Fun',
                  suffixes = c(".lapply",".for"))

sum(!is.na(Selftime$self.time.lapply))
sum(!is.na(Selftime$self.time.for))
Selftime[order(Selftime$self.time.lapply, decreasing = TRUE),
         c("Fun","self.time.lapply","self.time.for")]

Selftime[is.na(Selftime$self.time.for),]

More Related Contents:

  1. What are the performance differences between for-loops and the apply family of functions?
  2. Grouping functions (tapply, by, aggregate) and the *apply family
  3. Does ifelse really calculate both of its vectors every time? Is it slow?
  4. Is the “*apply” family really not vectorized?
  5. Access lapply index names inside FUN
  6. Split a large dataframe into a list of data frames based on common value in column
  7. Fastest way to replace NAs in a large data.table
  8. Read multiple CSV files into separate data frames
  9. Read all files in a folder and apply a function to each data frame
  10. .EACHI in data.table?
  11. Faster way to read fixed-width files
  12. Fast pairwise simple linear regression between variables in a data frame
  13. Returning anonymous functions from lapply – what is going wrong?
  14. Practical limits of R data frame
  15. Access and preserve list names in lapply function
  16. Fast Levenshtein distance in R?
  17. How to speed up subset by groups
  18. How can I prevent rbind() from geting really slow as dataframe grows larger?
  19. passing several arguments to FUN of lapply (and others *apply)
  20. How to index an element of a list object in R
  21. Efficiently computing a linear combination of data.table columns
  22. What methods can we use to reshape VERY large data sets?
  23. Writing multiple data frames into .csv files using R
  24. Using lapply to apply a function over list of data frames and saving output to files with different names
  25. Reason behind speed of fread in data.table package in R
  26. Creating a named list from two vectors (names, values)
  27. How to load data quickly into R?
  28. Union of intersecting vectors in a list in R
  29. How to read specific rows of CSV file with fread function
  30. Efficient string similarity grouping

Leave a Comment