Spark on yarn concept understanding

by

Adding to other answers.

  1. Is it necessary that spark is installed on all the nodes in the yarn
    cluster?

No, If the spark job is scheduling in YARN(either client or cluster mode). Spark installation is needed in many nodes only for standalone mode.

These are the visualizations of spark app deployment modes.

Spark Standalone Cluster

Spark standalone mode

In cluster mode driver will be sitting in one of the Spark Worker node whereas in client mode it will be within the machine which launched the job.

YARN cluster mode

YARN cluster mode

YARN client mode

YARN client mode

This table offers a concise list of differences between these modes:

differences among Standalone, YARN Cluster and YARN Client modes

pics source

  1. It says in the documentation “Ensure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client-side)
    configuration files for the Hadoop cluster”. Why does the client node have
    to install Hadoop when it is sending the job to cluster?

Hadoop installation is not mandatory but configurations(not all) are!. We can call them Gateway nodes. It’s for two main reasons.

  • The configuration contained in HADOOP_CONF_DIR directory will be distributed to the YARN cluster so that all containers used by the application use the same configuration.
  • In YARN mode the ResourceManager’s address is picked up from the
    Hadoop configuration(yarn-default.xml). Thus, the --master parameter is yarn.

Update: (2017-01-04)

Spark 2.0+ no longer requires a fat assembly jar for production
deployment. source

More Related Contents:

  1. Apache Spark: The number of cores vs. the number of executors
  2. Apache Hadoop Yarn – Underutilization of cores
  3. Where are logs in Spark on YARN?
  4. Spark iterate HDFS directory
  5. Begenner at spark Big data programming (spark code)
  6. Write to multiple outputs by key Spark – one Spark job
  7. Spark – load CSV file as DataFrame?
  8. How does Hadoop process records split across block boundaries?
  9. While writing to hdfs path getting error java.io.IOException: Failed to rename
  10. Read whole text files from a compression in Spark
  11. Container is running beyond memory limits
  12. How does Hadoop Namenode failover process works?
  13. Name node is in safe mode. Not able to leave
  14. How to fix corrupt HDFS FIles
  15. How to get ID of a map task in Spark?
  16. How can I access S3/S3n from a local Hadoop 2.6 installation?
  17. Pyspark: get list of files/directories on HDFS path
  18. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  19. How to access s3a:// files from Apache Spark?
  20. Is it better to have one large parquet file or lots of smaller parquet files?
  21. Namenode not getting started
  22. Easiest way to install Python dependencies on Spark executor nodes?
  23. Hive: Add partitions for existing folder structure
  24. Default Namenode port of HDFS is 50070.But I have come across at some places 8020 or 9000 [closed]
  25. How does Hadoop perform input splits?
  26. Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation
  27. Default number of reducers
  28. Difference between HBase and Hadoop/HDFS
  29. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  30. Cannot Read a file from HDFS using Spark

Leave a Comment