PySpark: java.lang.OutofMemoryError: Java heap space

by

After trying out loads of configuration parameters, I found that there is only one need to be changed to enable more Heap space and i.e. spark.driver.memory. 

sudo vim $SPARK_HOME/conf/spark-defaults.conf
#uncomment the spark.driver.memory and change it according to your use. I changed it to below
spark.driver.memory 15g
# press : and then wq! to exit vim editor

Close your existing spark application and re run it. You will not encounter this error again. 🙂

More Related Contents:

  1. How to deal with “java.lang.OutOfMemoryError: Java heap space” error?
  2. Error java.lang.OutOfMemoryError: GC overhead limit exceeded
  3. java.lang.OutOfMemoryError: Java heap space
  4. How to flatten a struct in a Spark dataframe?
  5. Spark: How to map Python with Scala or Java User Defined Functions?
  6. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  7. Spark Error – Unsupported class file major version
  8. How to set timezone to UTC in Apache Spark?
  9. How do I give Jenkins more heap space when it’s running as a daemon on Ubuntu?
  10. java.lang.OutOfMemoryError: Java heap space with NetBeans
  11. Running custom Java class in PySpark
  12. An out-of-memory error in clustering
  13. best Java code to control big heap memeory used (like 1G) [closed]
  14. How to do aggregate functions, may columns and extract back
  15. What is a StackOverflowError?
  16. Recommendations for a heap analysis tool for Java? [closed]
  17. Is a Java array of primitives stored in stack or heap?
  18. Can the JVM recover from an OutOfMemoryError without a restart
  19. java.lang.OutOfMemoryError: Java heap space in Maven
  20. Timestamp convert [duplicate]
  21. What is the largest possible heap size with a 64-bit JVM?
  22. Java: Unable to create new native thread
  23. How do I set Java’s min and max heap size through environment variables?
  24. How to view the current heap size that an application is using?
  25. Java OutOfMemoryError strange behaviour
  26. Java OutOfMemoryError in reading a large text file
  27. Steps in the memory allocation process for Java objects
  28. java.lang.OutOfMemoryError: unable to create new native thread error using ChromeDriver and Chrome through Selenium in Spring boot
  29. How to increase java heap size programmatically
  30. OutputStream OutOfMemoryError when sending HTTP

Leave a Comment