Customize SparkContext using sparkConf.set(..) when using spark-shell

by

Spark 2.0+

You should be able to use SparkSession.conf.set method to set some configuration option on runtime but it is mostly limited to SQL configuration.

Spark < 2.0

You can simply stop an existing context and create a new one:

import org.apache.spark.{SparkContext, SparkConf}

sc.stop()
val conf = new SparkConf().set("spark.executor.memory", "4g")
val sc = new SparkContext(conf)

As you can read in the official documentation:

Once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.

So as you can see stopping the context it is the only applicable option once shell has been started.

You can always use configuration files or --conf argument to spark-shell to set required parameters which will be used be the default context. In case of Kryo you should take a look at:

  • spark.kryo.classesToRegister
  • spark.kryo.registrator

See Compression and Serialization in Spark Configuration.

More Related Contents:

  1. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  2. How to define schema for custom type in Spark SQL?
  3. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  4. NullPointerException in Scala Spark, appears to be caused be collection type?
  5. What does setMaster `local[*]` mean in spark?
  6. Spark unionAll multiple dataframes
  7. How can I pass extra parameters to UDFs in Spark SQL?
  8. How to split a dataframe into dataframes with same column values?
  9. Spark performance for Scala vs Python
  10. How to use regex to include/exclude some input files in sc.textFile?
  11. How to define a custom aggregation function to sum a column of Vectors?
  12. Spark losing println() on stdout
  13. How to write spark streaming DF to Kafka topic
  14. Stackoverflow due to long RDD Lineage
  15. Better way to convert a string field into timestamp in Spark
  16. Spark UDAF with ArrayType as bufferSchema performance issues
  17. How to find spark RDD/Dataframe size?
  18. Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
  19. Provide schema while reading csv file as a dataframe in Scala Spark
  20. Explanation of fold method of spark RDD
  21. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  22. Replace null values in Spark DataFrame
  23. Spark RDD default number of partitions
  24. Why does Spark RDD partition has 2GB limit for HDFS?
  25. Calculate Cosine Similarity Spark Dataframe
  26. How to sort an RDD in Scala Spark?
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment