Why is the parallel package slower than just using apply?

by

Running jobs in parallel incurs overhead. Only if the jobs you fire at the worker nodes take a significant amount of time does parallelization improve overall performance. When the individual jobs take only milliseconds, the overhead of constantly firing off jobs will deteriorate overall performance. The trick is to divide the work over the nodes in such a way that the jobs are sufficiently long, say at least a few seconds. I used this to great effect running six Fortran models simultaneously, but these individual model runs took hours, almost negating the effect of overhead.

Note that I haven’t run your example, but the situation I describe above is often the issue when parallization takes longer than running sequentially.

More Related Contents:

  1. How can I print when using %dopar%
  2. run a for loop in parallel in R
  3. Saving multiple outputs of foreach dopar loop
  4. data.table and parallel computing
  5. Is there a vectorized parallel max() and min()?
  6. using parallel’s parLapply: unable to access variables within parallel code
  7. reading global variables using foreach in R
  8. How can I `print` or `cat` when using parallel
  9. parallel execution of random forest in R
  10. How can i create a contingency table in R? [closed]
  11. How do I use the aggregate function to mean|count|sum up multiple columns in R?
  12. Is R’s apply family more than syntactic sugar?
  13. Extract a dplyr tbl column as a vector
  14. dplyr mutate with conditional values
  15. Check for installed packages before running install.packages() [duplicate]
  16. Split up `…` arguments and distribute to multiple functions
  17. Linear Regression loop for each independent variable individually against dependent
  18. if-else vs ifelse with lists
  19. Code chunk font size in Rmarkdown with knitr and latex
  20. Mass variable declaration and assignment in R?
  21. How to replace NA with most recent non-NA by group? [duplicate]
  22. R Shiny set DataTable column width
  23. R Reading in a zip data file without unzipping it
  24. Use loop to generate section of text in rmarkdown
  25. Add a column with count of NAs and Mean
  26. Define dimensions of an empty dataframe
  27. LDA with topicmodels, how can I see which topics different documents belong to?
  28. How to add Rtools\bin to the system path in R
  29. Merge dataframes, different lengths
  30. Filtering rows in R unexpectedly removes NAs when using subset or dplyr::filter

Leave a Comment