Overwrite specific partitions in spark dataframe write method

by

Finally! This is now a feature in Spark 2.3.0:
SPARK-20236

To use it, you need to set the spark.sql.sources.partitionOverwriteMode setting to dynamic, the dataset needs to be partitioned, and the write mode overwrite. Example:

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
data.write.mode("overwrite").insertInto("partitioned_table")

I recommend doing a repartition based on your partition column before writing, so you won’t end up with 400 files per folder.

Before Spark 2.3.0, the best solution would be to launch SQL statements to delete those partitions and then write them with mode append.

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. How to import multiple csv files in a single load?
  3. How to save/insert each DStream into a permanent table
  4. Find maximum row per group in Spark DataFrame
  5. Split Spark Dataframe string column into multiple columns
  6. While writing to hdfs path getting error java.io.IOException: Failed to rename
  7. Multiple Aggregate operations on the same column of a spark dataframe
  8. Partitioning in spark while reading from RDBMS via JDBC
  9. How to control partition size in Spark SQL
  10. Avoid performance impact of a single partition mode in Spark window functions
  11. How to access element of a VectorUDT column in a Spark DataFrame?
  12. Spark DataFrame: count distinct values of every column
  13. Python Spark Cumulative Sum by Group Using DataFrame
  14. How to split a list to multiple columns in Pyspark?
  15. pyspark dataframe filter or include based on list
  16. Spark parquet partitioning : Large number of files
  17. What is the maximum size for a broadcast object in Spark?
  18. How to find mean of grouped Vector columns in Spark SQL?
  19. Generate a Spark StructType / Schema from a case class
  20. How does createOrReplaceTempView work in Spark?
  21. Fill in null with previously known good value with pyspark
  22. PySpark – get row number for each row in a group
  23. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  24. Apache spark dealing with case statements
  25. pyspark: count distinct over a window
  26. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  27. What does “Correlated scalar subqueries must be Aggregated” mean?
  28. How to improve broadcast Join speed with between condition in Spark
  29. Spark SQL referencing attributes of UDT
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment