Spark – Error “A master URL must be set in your configuration” when submitting an app

by

The TLDR:

.config("spark.master", "local")

a list of the options for spark.master in spark 2.2.1

I ended up on this page after trying to run a simple Spark SQL java program in local mode. To do this, I found that I could set spark.master using:

SparkSession spark = SparkSession
.builder()
.appName("Java Spark SQL basic example")
.config("spark.master", "local")
.getOrCreate();

An update to my answer:

To be clear, this is not what you should do in a production environment. In a production environment, spark.master should be specified in one of a couple other places: either in $SPARK_HOME/conf/spark-defaults.conf (this is where cloudera manager will put it), or on the command line when you submit the app. (ex spark-submit –master yarn).

If you specify spark.master to be ‘local’ in this way, spark will try to run in a single jvm, as indicated by the comments below. If you then try to specify –deploy-mode cluster, you will get an error ‘Cluster deploy mode is not compatible with master “local”‘. This is because setting spark.master=local means that you are NOT running in cluster mode.

Instead, for a production app, within your main function (or in functions called by your main function), you should simply use: 

SparkSession
.builder()
.appName("Java Spark SQL basic example")
.getOrCreate();

This will use the configurations specified on the command line/in config files.

Also, to be clear on this too: –master and “spark.master” are the exact same parameter, just specified in different ways. Setting spark.master in code, like in my answer above, will override attempts to set –master, and will override values in spark-defaults.conf, so don’t do it in production. Its great for tests though.

also, see this answer.
which links to a list of the options for spark.master and what each one actually does.

a list of the options for spark.master in spark 2.2.1

More Related Contents:

  1. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  2. Write to multiple outputs by key Spark – one Spark job
  3. How does HashPartitioner work?
  4. Spark 2.0 Dataset vs DataFrame
  5. Spark – load CSV file as DataFrame?
  6. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  7. How to define partitioning of DataFrame?
  8. Case class equality in Apache Spark
  9. Why accesing DataFrame from UDF results in NullPointerException?
  10. Dropping a nested column from Spark DataFrame
  11. How to read records in JSON format from Kafka using Structured Streaming?
  12. Spark dataframe write method writing many small files
  13. Filling gaps in timeseries Spark
  14. Reading DataFrame from partitioned parquet file
  15. java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags
  16. How to use COGROUP for large datasets
  17. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  18. Parsing multiline records in Scala
  19. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  20. Create new Dataframe with empty/null field values
  21. Perform a typed join in Scala with Spark Datasets
  22. Left Anti join in Spark?
  23. DataFrame-ified zipWithIndex
  24. Spark: Add column to dataframe conditionally
  25. Explanation of fold method of spark RDD
  26. How to explode an array into multiple columns in Spark
  27. Spark : how to run spark file from spark shell
  28. Spark Dataframe Random UUID changes after every transformation/action
  29. How to set hadoop configuration values from pyspark
  30. Spark ML VectorAssembler returns strange output

Leave a Comment