How to overwrite the output directory in spark

by

UPDATE: Suggest using Dataframes, plus something like ... .write.mode(SaveMode.Overwrite) ....

Handy pimp:

implicit class PimpedStringRDD(rdd: RDD[String]) {
    def write(p: String)(implicit ss: SparkSession): Unit = {
      import ss.implicits._
      rdd.toDF().as[String].write.mode(SaveMode.Overwrite).text(p)
    }
  }

For older versions try

yourSparkConf.set("spark.hadoop.validateOutputSpecs", "false")
val sc = SparkContext(yourSparkConf)

In 1.1.0 you can set conf settings using the spark-submit script with the –conf flag.

WARNING (older versions): According to @piggybox there is a bug in Spark where it will only overwrite files it needs to to write it’s part- files, any other files will be left unremoved.

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Find maximum row per group in Spark DataFrame
  3. Integrating Spark Structured Streaming with the Confluent Schema Registry
  4. What is the difference between map and flatMap and a good use case for each?
  5. Python Spark Cumulative Sum by Group Using DataFrame
  6. How to split a list to multiple columns in Pyspark?
  7. Link Spark with iPython Notebook
  8. Default Partitioning Scheme in Spark
  9. How to manually set group.id and commit kafka offsets in spark structured streaming?
  10. Spark parquet partitioning : Large number of files
  11. How do you control the size of the output file?
  12. Any performance issues forcing eager evaluation using count in spark?
  13. What is a task in Spark? How does the Spark worker execute the jar file?
  14. Debugging “Managed memory leak detected” in Spark 1.6.0
  15. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  16. Spark: how to get the number of written rows?
  17. Why are Spark Parquet files for an aggregate larger than the original?
  18. Apache spark dealing with case statements
  19. What’s the difference between Spark ML and MLLIB packages
  20. pyspark: count distinct over a window
  21. Mind blown: RDD.zip() method
  22. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  23. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  24. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  25. reduce result datasets into single dataset
  26. How to exclude multiple columns in Spark dataframe in Python
  27. Why does Spark job fail with “too many open files”?
  28. How does Distinct() function work in Spark?
  29. Stratified sampling with pyspark
  30. Get CSV to Spark dataframe

Leave a Comment