How to change memory per node for apache spark worker

by

When using 1.0.0+ and using spark-shell or spark-submit, use the --executor-memory option. E.g.

spark-shell --executor-memory 8G ...

0.9.0 and under:

When you start a job or start the shell change the memory. We had to modify the spark-shell script so that it would carry command line arguments through as arguments for the underlying java application. In particular:

OPTIONS="$@"
...
$FWDIR/bin/spark-class $OPTIONS org.apache.spark.repl.Main "$@"

Then we can run our spark shell as follows:

spark-shell -Dspark.executor.memory=6g

When configuring it for a standalone jar, I set the system property programmatically before creating the spark context and pass the value in as a command line argument (I can make it shorter than the long winded system props then).

System.setProperty("spark.executor.memory", valueFromCommandLine)

As for changing the default cluster wide, sorry, not entirely sure how to do it properly.

One final point – I’m a little worried by the fact you have 2 nodes with 2GB and one with 6GB. The memory you can use will be limited to the smallest node – so here 2GB.

More Related Contents:

  1. How to set Apache Spark Executor memory
  2. Purpose of memory alignment
  3. How does a “stack overflow” occur and how do you prevent it?
  4. Determining memory usage of objects?
  5. Cannot free memory once occupied by bytes.Buffer
  6. iOS6 MKMapView using a ton of memory, to the point of crashing the app, anyone else notice this?
  7. How to determine maximum stack usage?
  8. How much faster is the memory usually than the disk?
  9. PHP Warning: POST Content-Length of 113 bytes exceeds the limit of -1988100096 bytes in Unknown
  10. How to determine needed memory of Keras model?
  11. What is the best way to add a value to an array in state
  12. Go 1.3 Garbage collector not releasing server memory back to system
  13. What’s the relationship between “a” heap and “the” heap?
  14. How do cache lines work?
  15. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  16. How to assign more memory to Netbeans?
  17. Difference between word addressable and byte addressable
  18. Sort with the limited memory
  19. Pass large amount of binary data from u-boot to linux kernel
  20. intrinsic memcmp
  21. Memory alignment on a 32-bit Intel processor
  22. GCC’s reordering of read/write instructions
  23. Spring Boot and multiple external configuration files
  24. Why does a JVM report more committed memory than the linux process resident set size?
  25. Pointer address in a C multidimensional array
  26. What is the default stack size, can it grow, how does it work with garbage collection?
  27. spark.sql.crossJoin.enabled for Spark 2.x
  28. What’s the differences between VirtualAlloc and HeapAlloc?
  29. Malloc vs custom allocator: Malloc has a lot of overhead. Why?
  30. Determining Stack Space with Visual Studio

Leave a Comment