Debugging “Managed memory leak detected” in Spark 1.6.0

by

The short answer is that users are not supposed to see this message. Users are not supposed to be able to create memory leaks in the unified memory manager.

That such leaks happen is a Spark bug: SPARK-11293

But if you want to understand the cause of a memory leak, this is how I did it.

  1. Download the Spark source code and make sure you can build it and your build works.
  2. In TaskMemoryManager.java add extra logging in acquireExecutionMemory and releaseExecutionMemory: logger.error("stack trace:", new Exception());
  3. Change all the other debug logs to error in TaskMemoryManager.java. (Easier than figuring out logging configurations…)

Now you will see the full stack trace for all allocations and deallocations. Try to match them up and find the allocations without deallocations. You now have the stack trace for the source of the leak.

More Related Contents:

  1. Find maximum row per group in Spark DataFrame
  2. Partitioning in spark while reading from RDBMS via JDBC
  3. What is the difference between map and flatMap and a good use case for each?
  4. Best Practice to launch Spark Applications via Web Application?
  5. How to save/insert each DStream into a permanent table
  6. How DAG works under the covers in RDD?
  7. Spark yarn cluster vs client – how to choose which one to use?
  8. Spark on YARN + Secured hbase
  9. Spark load data and add filename as dataframe column
  10. PySpark: how to resample frequencies
  11. Any performance issues forcing eager evaluation using count in spark?
  12. Explode array data into rows in spark [duplicate]
  13. Spark: how to get the number of written rows?
  14. ALS model – how to generate full_u * v^t * v?
  15. PySpark – get row number for each row in a group
  16. Why are Spark Parquet files for an aggregate larger than the original?
  17. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  18. Apache spark dealing with case statements
  19. pyspark: count distinct over a window
  20. Mind blown: RDD.zip() method
  21. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  22. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  23. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  24. What do the numbers on the progress bar mean in spark-shell?
  25. pyspark: rolling average using timeseries data
  26. What are broadcast variables? What problems do they solve?
  27. What does “Correlated scalar subqueries must be Aggregated” mean?
  28. How to improve broadcast Join speed with between condition in Spark
  29. Spark ALS predictAll returns empty
  30. Spark SQL referencing attributes of UDT

Leave a Comment