Create new Dataframe with empty/null field values

by

It is possible to use lit(null):

import org.apache.spark.sql.functions.{lit, udf}

case class Record(foo: Int, bar: String)
val df = Seq(Record(1, "foo"), Record(2, "bar")).toDF

val dfWithFoobar = df.withColumn("foobar", lit(null: String))

One problem here is that the column type is null:

scala> dfWithFoobar.printSchema
root
 |-- foo: integer (nullable = false)
 |-- bar: string (nullable = true)
 |-- foobar: null (nullable = true)

and it is not retained by the csv writer. If it is a hard requirement you can cast column to the specific type (lets say String), with either DataType

import org.apache.spark.sql.types.StringType

df.withColumn("foobar", lit(null).cast(StringType))

or string description

df.withColumn("foobar", lit(null).cast("string"))

or use an UDF like this:

val getNull = udf(() => None: Option[String]) // Or some other type

df.withColumn("foobar", getNull()).printSchema
root
 |-- foo: integer (nullable = false)
 |-- bar: string (nullable = true)
 |-- foobar: string (nullable = true)

A Python equivalent can be found here: Add an empty column to spark DataFrame

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. How to zip two (or more) DataFrame in Spark
  11. Joining Spark dataframes on the key
  12. Apache Spark how to append new column from list/array to Spark dataframe
  13. Provide schema while reading csv file as a dataframe in Scala Spark
  14. Renaming column names of a DataFrame in Spark Scala
  15. Derive multiple columns from a single column in a Spark DataFrame
  16. Replace missing values with mean – Spark Dataframe
  17. What is going wrong with `unionAll` of Spark `DataFrame`?
  18. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  19. How to get keys and values from MapType column in SparkSQL DataFrame
  20. Spark DataFrames when udf functions do not accept large enough input variables
  21. Filter spark DataFrame on string contains
  22. How to avoid duplicate columns after join?
  23. Flattening Rows in Spark
  24. How to flatmap a nested Dataframe in Spark
  25. How to write unit tests in Spark 2.0+?
  26. how to filter out a null value from spark dataframe
  27. What are the various join types in Spark?
  28. How to calculate the size of dataframe in bytes in Spark?
  29. Spark-Monotonically increasing id not working as expected in dataframe?
  30. Spark Dataframe Random UUID changes after every transformation/action

Leave a Comment