Add Number of days column to Date Column in same dataframe for Spark Scala App

by

No need to use an UDF, you can do it using an SQL expression:

val newDF = df.withColumn("new_date", expr("date_add(current_date,days)"))

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Append a column to Data Frame in Apache Spark 1.3
  9. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  10. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  11. How to zip two (or more) DataFrame in Spark
  12. Joining Spark dataframes on the key
  13. Apache Spark how to append new column from list/array to Spark dataframe
  14. Create new Dataframe with empty/null field values
  15. Provide schema while reading csv file as a dataframe in Scala Spark
  16. Renaming column names of a DataFrame in Spark Scala
  17. Derive multiple columns from a single column in a Spark DataFrame
  18. Replace missing values with mean – Spark Dataframe
  19. What is going wrong with `unionAll` of Spark `DataFrame`?
  20. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  21. Replace null values in Spark DataFrame
  22. How to get keys and values from MapType column in SparkSQL DataFrame
  23. Spark Dataframe Random UUID changes after every transformation/action
  24. Spark DataFrames when udf functions do not accept large enough input variables
  25. Filter spark DataFrame on string contains
  26. Automatically and Elegantly flatten DataFrame in Spark SQL
  27. How do I skip a header from CSV files in Spark?
  28. Spark / Scala: forward fill with last observation
  29. Apache Spark: Get number of records per partition
  30. How to save a spark DataFrame as csv on disk?

Leave a Comment