Spark Dataframes UPSERT to Postgres Table

by

It is not supported. DataFrameWriter can either append to or overwrite existing table. If your application requires more complex logic you’ll have to deal with this manually.

One option is to use an action (foreach, foreachPartition) with standard JDBC connection. Another one is to write to a temporary and handle the rest directly in the database.

See also SPARK-19335 (Spark should support doing an efficient DataFrame Upsert via JDBC) and related proposals.

More Related Contents:

  1. How to select the first row of each group?
  2. Querying Spark SQL DataFrame with complex types
  3. How to query JSON data column using Spark DataFrames?
  4. How to define partitioning of DataFrame?
  5. How do I detect if a Spark DataFrame has a column
  6. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  7. How to split a dataframe into dataframes with same column values?
  8. How to create an empty DataFrame with a specified schema?
  9. Provide schema while reading csv file as a dataframe
  10. Dropping a nested column from Spark DataFrame
  11. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  12. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  13. How to zip two (or more) DataFrame in Spark
  14. Joining Spark dataframes on the key
  15. Apache Spark how to append new column from list/array to Spark dataframe
  16. Create new Dataframe with empty/null field values
  17. Provide schema while reading csv file as a dataframe in Scala Spark
  18. Renaming column names of a DataFrame in Spark Scala
  19. Derive multiple columns from a single column in a Spark DataFrame
  20. Replace missing values with mean – Spark Dataframe
  21. What is going wrong with `unionAll` of Spark `DataFrame`?
  22. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  23. How to get keys and values from MapType column in SparkSQL DataFrame
  24. Spark DataFrames when udf functions do not accept large enough input variables
  25. Filter spark DataFrame on string contains
  26. Difference between === null and isNull in Spark DataDrame
  27. Difference between DataFrame, Dataset, and RDD in Spark
  28. Requirements for converting Spark dataframe to Pandas/R dataframe
  29. Spark Strutured Streaming automatically converts timestamp to local time
  30. Spark sql queries vs dataframe functions

Leave a Comment