Installing of SparkR

by

You can install directly from a GitHub repository:

if (!require('devtools')) install.packages('devtools')
devtools::install_github('apache/[email protected]', subdir="R/pkg")

You should choose tag (v2.x.x above) corresponding to the version of Spark you use. You can find a full list of tags on the project page or directly from R using GitHub API:

jsonlite::fromJSON("https://api.github.com/repos/apache/spark/tags")$name

If you’ve downloaded binary package from a downloads page R library is in a R/lib/SparkR subdirectory. It can be used to install SparkR directly. For example:

$ export SPARK_HOME=/path/to/spark/directory
$ cd $SPARK_HOME/R/pkg/
$ R -e "devtools::install('.')"

You can also add R lib to .libPaths (taken from here):

Sys.setenv(SPARK_HOME='/path/to/spark/directory')
.libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'), .libPaths()))

Finally, you can use sparkR shell without any additional steps:

$ /path/to/spark/directory/bin/sparkR

Edit

According to Spark 2.1.0 Release Notes should be available on CRAN in the future:

Standalone installable package built with the Apache Spark release. We will be submitting this to CRAN soon.

You can follow SPARK-15799 to check the progress.

Edit 2

While SPARK-15799 has been merged, satisfying CRAN requirements proved to be challenging (see for example discussions about 2.2.2, 2.3.1, 2.4.0), and the packages has been subsequently removed (see for example SparkR was removed from CRAN on 2018-05-01, CRAN SparkR package removed?). As the result methods listed in the original post are still the most reliable solutions.

Edit 3

OK, SparkR is back up on CRAN again, v2.4.1. install.packages('SparkR') should work again (it may take a couple of days for the mirrors to reflect this)

More Related Contents:

  1. Sparklyr: how to center a Spark table based on column?
  2. How to make plots from distributed data from R
  3. Painless way to install a new version of R?
  4. How to use an image as a point in ggplot?
  5. How to merge 2 vectors alternating indexes?
  6. How to put labels over geom_bar in R with ggplot2
  7. Wrap long axis labels via labeller=label_wrap in ggplot2
  8. How to efficiently calculate distance between pair of coordinates using data.table :=
  9. Access variable value where the name of variable is stored in a string
  10. R Shiny – add tabPanel to tabsetPanel dynamically (with the use of renderUI)
  11. How do I change the formatting of numbers on an axis with ggplot? [duplicate]
  12. Displaying a greater than or equal sign
  13. How to change order of boxplots when using ggplot2?
  14. R package lattice won’t plot if run using source()
  15. Modifying fonts in ggplot2
  16. listing contents of an R data file without loading
  17. How do you determine the namespace of a function?
  18. How to fill with different colors between two lines? (originally: fill geom_polygon with different colors above and below y = 0 (or any other value)?)
  19. How to extract certain columns from a list of data frames
  20. How to find out which package version is loaded in R?
  21. ggplot geom_point() with colors based on specific, discrete values
  22. Moving color key in R heatmap.2 (function of gplots package)
  23. GGplot custom scale transformation with custom ticks
  24. How to increase the size of points in legend of ggplot2? [duplicate]
  25. Creating a named list from two vectors (names, values)
  26. Are there global variables in R Shiny?
  27. Shifting a column down by one
  28. How to change stacking order in stacked bar chart in R?
  29. How can I count the number of NAs per group?
  30. R: How to split a data frame into training, validation, and test sets?

Leave a Comment