How to set Apache Spark Executor memory

by

Since you are running Spark in local mode, setting spark.executor.memory won’t have any effect, as you have noticed. The reason for this is that the Worker “lives” within the driver JVM process that you start when you start spark-shell and the default memory used for that is 512M. You can increase that by setting spark.driver.memory to something higher, for example 5g. You can do that by either:

  • setting it in the properties file (default is $SPARK_HOME/conf/spark-defaults.conf), 

    spark.driver.memory              5g

  • or by supplying configuration setting at runtime

    $ ./bin/spark-shell --driver-memory 5g

Note that this cannot be achieved by setting it in the application, because it is already too late by then, the process has already started with some amount of memory.

The reason for 265.4 MB is that Spark dedicates spark.storage.memoryFraction * spark.storage.safetyFraction to the total amount of storage memory and by default they are 0.6 and 0.9.

512 MB * 0.6 * 0.9 ~ 265.4 MB

So be aware that not the whole amount of driver memory will be available for RDD storage.

But when you’ll start running this on a cluster, the spark.executor.memory setting will take over when calculating the amount to dedicate to Spark’s memory cache.

More Related Contents:

  1. How to change memory per node for apache spark worker
  2. Purpose of memory alignment
  3. How does a “stack overflow” occur and how do you prevent it?
  4. Determining memory usage of objects?
  5. Cannot free memory once occupied by bytes.Buffer
  6. iOS6 MKMapView using a ton of memory, to the point of crashing the app, anyone else notice this?
  7. How to determine maximum stack usage?
  8. How much faster is the memory usually than the disk?
  9. PHP Warning: POST Content-Length of 113 bytes exceeds the limit of -1988100096 bytes in Unknown
  10. How to determine needed memory of Keras model?
  11. What is the best way to add a value to an array in state
  12. Go 1.3 Garbage collector not releasing server memory back to system
  13. What’s the relationship between “a” heap and “the” heap?
  14. How do cache lines work?
  15. How to assign more memory to Netbeans?
  16. Difference between word addressable and byte addressable
  17. Sort with the limited memory
  18. Pass large amount of binary data from u-boot to linux kernel
  19. intrinsic memcmp
  20. Memory alignment on a 32-bit Intel processor
  21. GCC’s reordering of read/write instructions
  22. How to make plots from distributed data from R
  23. Keep persistent variables in memory between runs of Python script
  24. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  25. How to save memory when reading a file in Php?
  26. DataFrame partitionBy to a single Parquet file (per partition)
  27. Temp table caching with spark-sql
  28. Why does Spark RDD partition has 2GB limit for HDFS?
  29. How can I connect to a postgreSQL database into Apache Spark using scala?
  30. Garbage Collection not happening even when needed

Leave a Comment