Overwrite only some partitions in a partitioned spark Dataset

by

Since Spark 2.3.0 this is an option when overwriting a table. To overwrite it, you need to set the new spark.sql.sources.partitionOverwriteMode setting to dynamic, the dataset needs to be partitioned, and the write mode overwrite.
Example in scala:

spark.conf.set(
  "spark.sql.sources.partitionOverwriteMode", "dynamic"
)
data.write.mode("overwrite").insertInto("partitioned_table")

I recommend doing a repartition based on your partition column before writing, so you won’t end up with 400 files per folder.

Before Spark 2.3.0, the best solution would be to launch SQL statements to delete those partitions and then write them with mode append.

More Related Contents:

  1. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  2. How to control partition size in Spark SQL
  3. How to optimize partitioning when migrating data from JDBC source?
  4. How to connect to remote hive server from spark [duplicate]
  5. Use collect_list and collect_set in Spark SQL
  6. Multiple Spark applications with HiveContext
  7. What is the difference between Apache Spark SQLContext vs HiveContext?
  8. Array Intersection in Spark SQL
  9. spark createOrReplaceTempView vs createGlobalTempView
  10. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  11. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  12. How to calculate Median in spark sqlContext for column of data type double
  13. Spark Dataframe validating column names for parquet writes
  14. Spark: subtract two DataFrames
  15. TypeError: Column is not iterable – How to iterate over ArrayType()?
  16. How to overwrite the output directory in spark
  17. Is groupByKey ever preferred over reduceByKey
  18. Pyspark : forward fill with last observation for a DataFrame
  19. spark.sql.crossJoin.enabled for Spark 2.x
  20. Why do I have to explicitly tell Spark what to cache?
  21. multiple conditions for filter in spark data frames
  22. How to pass whole Row to UDF – Spark DataFrame filter
  23. Rename more than one column using withColumnRenamed
  24. Spark lists all leaf node even in partitioned data
  25. AWS EMR – ModuleNotFoundError: No module named ‘pyarrow’
  26. How to exclude multiple columns in Spark dataframe in Python
  27. Why does Spark job fail with “too many open files”?
  28. How does Distinct() function work in Spark?
  29. Stratified sampling with pyspark
  30. Get CSV to Spark dataframe

Leave a Comment