Which cluster type should I choose for Spark? [closed]

by

I think the best to answer that are those who work on Spark. So, from Learning Spark

Start with a standalone cluster if this is a new deployment.
Standalone mode is the easiest to set up and will provide almost all
the same features as the other cluster managers if you are only
running Spark.

If you would like to run Spark alongside other applications, or to use
richer resource scheduling capabilities (e.g. queues), both YARN and
Mesos provide these features. Of these, YARN will likely be
preinstalled in many Hadoop distributions.

One advantage of Mesos over both YARN and standalone mode is its
fine-grained sharing option, which lets interactive applications such
as the Spark shell scale down their CPU allocation between commands.
This makes it attractive in environments where multiple users are
running interactive shells.

In all cases, it is best to run Spark on the same nodes as HDFS for
fast access to storage. You can install Mesos or the standalone
cluster manager on the same nodes manually, or most Hadoop
distributions already install YARN and HDFS together.

More Related Contents:

  1. Spark Standalone Number Executors/Cores Control
  2. Spark yarn cluster vs client – how to choose which one to use?
  3. Apache Spark: setting executor instances does not change the executors
  4. Apache Spark: Differences between client and cluster deploy modes
  5. Using a column value as a parameter to a spark DataFrame function
  6. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  7. What does “Stage Skipped” mean in Apache Spark web UI?
  8. How to import multiple csv files in a single load?
  9. How are stages split into tasks in Spark?
  10. How to tune spark executor number, cores and executor memory?
  11. Which operations preserve RDD order?
  12. How to split a list to multiple columns in Pyspark?
  13. Apache Spark does not delete temporary directories
  14. Spark parquet partitioning : Large number of files
  15. How do you control the size of the output file?
  16. What is the maximum size for a broadcast object in Spark?
  17. Efficient pyspark join
  18. How does createOrReplaceTempView work in Spark?
  19. ALS model – how to generate full_u * v^t * v?
  20. PySpark – get row number for each row in a group
  21. Save ML model for future usage
  22. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  23. Overwrite only some partitions in a partitioned spark Dataset
  24. Using windowing functions in Spark
  25. How to assign unique contiguous numbers to elements in a Spark RDD
  26. What is spark.driver.maxResultSize?
  27. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  28. How to get Kafka offsets for structured query for manual and reliable offset management?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment