Apache Spark does not delete temporary directories

by

Three SPARK_WORKER_OPTS exists to support the worker application folder cleanup, copied here for further reference: from Spark Doc

  • spark.worker.cleanup.enabled, default value is false, Enable periodic cleanup of worker / application directories. Note that this only affects standalone mode, as YARN works differently. Only the directories of stopped applications are cleaned up.

  • spark.worker.cleanup.interval, default is 1800, i.e. 30 minutes, Controls the interval, in seconds, at which the worker cleans up old application work dirs on the local machine.

  • spark.worker.cleanup.appDataTtl, default is 7*24*3600 (7 days), The number of seconds to retain application work directories on each worker. This is a Time To Live and should depend on the amount of available disk space you have. Application logs and jars are downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, especially if you run jobs very frequently.

More Related Contents:

  1. Spark Dataframe validating column names for parquet writes
  2. How to optimize partitioning when migrating data from JDBC source?
  3. Spark: subtract two DataFrames
  4. TypeError: Column is not iterable – How to iterate over ArrayType()?
  5. How to overwrite the output directory in spark
  6. Is groupByKey ever preferred over reduceByKey
  7. How DAG works under the covers in RDD?
  8. Spark on YARN + Secured hbase
  9. Spark DataFrame Schema Nullable Fields
  10. PySpark: how to resample frequencies
  11. Explode array data into rows in spark [duplicate]
  12. Why does Spark think this is a cross / Cartesian join
  13. Why does sortBy transformation trigger a Spark job?
  14. Spark SQL broadcast hash join
  15. What is the difference between cache and persist?
  16. What is the difference between Apache Spark SQLContext vs HiveContext?
  17. Saving dataframe to local file system results in empty results
  18. Dealing with a large gzipped file in Spark
  19. When are accumulators truly reliable?
  20. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  21. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  22. Spark lists all leaf node even in partitioned data
  23. How to get day of week in SparkSQL?
  24. AWS EMR – ModuleNotFoundError: No module named ‘pyarrow’
  25. Array Intersection in Spark SQL
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. How to optimize shuffle spill in Apache Spark application
  28. How to calculate Median in spark sqlContext for column of data type double
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Understanding Spark’s caching

Leave a Comment