Apache Spark: Differences between client and cluster deploy modes

by

What are the practical differences between Spark Standalone client
deploy mode and cluster deploy mode? What are the pro’s and con’s of
using each one?

Let’s try to look at the differences between client and cluster mode.

Client:

  • Driver runs on a dedicated server (Master node) inside a dedicated process. This means it has all available resources at it’s disposal to execute work.
  • Driver opens up a dedicated Netty HTTP server and distributes the JAR files specified to all Worker nodes (big advantage).
  • Because the Master node has dedicated resources of it’s own, you don’t need to “spend” worker resources for the Driver program.
  • If the driver process dies, you need an external monitoring system to reset it’s execution.

Cluster:

  • Driver runs on one of the cluster’s Worker nodes. The worker is chosen by the Master leader
  • Driver runs as a dedicated, standalone process inside the Worker.
  • Driver programs takes up at least 1 core and a dedicated amount of memory from one of the workers (this can be configured).
  • Driver program can be monitored from the Master node using the --supervise flag and be reset in case it dies.
  • When working in Cluster mode, all JARs related to the execution of your application need to be publicly available to all the workers. This means you can either manually place them in a shared place or in a folder for each of the workers.

Which one is better? Not sure, that’s actually for you to experiment and decide. This is no better decision here, you gain things from the former and latter, it’s up to you to see which one works better for your use-case.

How to I choose which one my application is going to be running on,
using spark-submit

The way to choose which mode to run in is by using the --deploy-mode flag. From the Spark Configuration page:

/bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

More Related Contents:

  1. Spark Standalone Number Executors/Cores Control
  2. Which cluster type should I choose for Spark? [closed]
  3. How to melt Spark DataFrame?
  4. Spark SQL: apply aggregate functions to a list of columns
  5. Unpivot in spark-sql/pyspark
  6. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  7. How does Spark partition(ing) work on files in HDFS?
  8. How to check if spark dataframe is empty?
  9. Spark: subtract two DataFrames
  10. Spark Transformation – Why is it lazy and what is the advantage?
  11. Xml processing in Spark
  12. What are workers, executors, cores in Spark Standalone cluster?
  13. How to connect to remote hive server from spark [duplicate]
  14. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  15. Adding a group count column to a PySpark dataframe
  16. Multiple Spark applications with HiveContext
  17. NoClassDefFoundError com.apache.hadoop.fs.FSDataInputStream when execute spark-shell
  18. Spark: Best practice for retrieving big data from RDD to local machine
  19. Spark-Obtaining file name in RDDs
  20. How do I add an persistent column of row ids to Spark DataFrame?
  21. How does Spark aggregate function – aggregateByKey work?
  22. Convert null values to empty array in Spark DataFrame
  23. DataFrame partitionBy to a single Parquet file (per partition)
  24. Temp table caching with spark-sql
  25. How to assign unique contiguous numbers to elements in a Spark RDD
  26. What is spark.driver.maxResultSize?
  27. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  28. Stratified sampling with pyspark
  29. How to use groupBy to collect rows into a map?
  30. When to use Spark DataFrame/Dataset API and when to use plain RDD?

Leave a Comment