Replace null values in Spark DataFrame

by

This is basically very simple. You’ll need to create a new DataFrame. I’m using the DataFrame df that you have defined earlier.

val newDf = df.na.fill("e",Seq("blank"))

DataFrames are immutable structures.
Each time you perform a transformation which you need to store, you’ll need to affect the transformed DataFrame to a new value.

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Append a column to Data Frame in Apache Spark 1.3
  9. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  10. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  11. How to zip two (or more) DataFrame in Spark
  12. Joining Spark dataframes on the key
  13. Apache Spark how to append new column from list/array to Spark dataframe
  14. Create new Dataframe with empty/null field values
  15. Provide schema while reading csv file as a dataframe in Scala Spark
  16. Renaming column names of a DataFrame in Spark Scala
  17. Derive multiple columns from a single column in a Spark DataFrame
  18. Replace missing values with mean – Spark Dataframe
  19. What is going wrong with `unionAll` of Spark `DataFrame`?
  20. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  21. How to get keys and values from MapType column in SparkSQL DataFrame
  22. Add Number of days column to Date Column in same dataframe for Spark Scala App
  23. Spark Dataframe Random UUID changes after every transformation/action
  24. Spark DataFrames when udf functions do not accept large enough input variables
  25. Filter spark DataFrame on string contains
  26. Case class equality in Apache Spark
  27. Filling gaps in timeseries Spark
  28. Spark: Add column to dataframe conditionally
  29. Explanation of fold method of spark RDD
  30. Difference between === null and isNull in Spark DataDrame

Leave a Comment