Retain keys with null values while writing JSON in spark

by

Apparently, spark does not provide any option to handle nulls. So following custom solution should work.

import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
import com.fasterxml.jackson.databind.ObjectMapper

case class EventHeader(accept_language:String,app_id:String,app_name:String,client_ip_address:String,event_id: String,event_timestamp:String,offering_id:String,server_ip_address:String,server_timestamp:Long,topic_name:String,version:String)

val ds = Seq(EventHeader(null,"App_ID",null,"IP","ID",null,"Offering","IP",1492565987565L,"Topic","1.0")).toDS()

val ds1 = ds.mapPartitions(records => {
val mapper = new ObjectMapper with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
records.map(mapper.writeValueAsString(_))
})

ds1.coalesce(1).write.text("hdfs://localhost:9000/user/dedupe_employee")

This will produce output as :

{"accept_language":null,"app_id":"App_ID","app_name":null,"client_ip_address":"IP","event_id":"ID","event_timestamp":null,"offering_id":"Offering","server_ip_address":"IP","server_timestamp":1492565987565,"topic_name":"Topic","version":"1.0"}

More Related Contents:

  1. How to handle this scenario in spark? [closed]
  2. How to do aggregate functions, may columns and extract back
  3. How to flatten a struct in a Spark dataframe?
  4. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  5. Spark sql how to explode without losing null values
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. spark broadcast variable Map giving null value
  8. How to set timezone to UTC in Apache Spark?
  9. Change output filename prefix for DataFrame.write()
  10. How do I call a UDF on a Spark DataFrame using JAVA?
  11. How to use Column.isin in Java?
  12. Why does SparkSession execute twice for one action?
  13. Gson: How to exclude specific fields from Serialization without annotations
  14. How should I escape strings in JSON?
  15. Difference between JSONObject and JSONArray
  16. Serializing with Jackson (JSON) – getting “No serializer found”?
  17. POST request via RestTemplate in JSON
  18. Convert JSON String to Pretty Print JSON output using Jackson
  19. Sending a JSON HTTP POST request from Android
  20. How to make an Action to accept dynamic JSON data from the user interface in Struts 2?
  21. How do I obtain the Jackson ObjectMapper in use by Spring 4.1?
  22. What’s the best way to load a JSONObject from a json text file?
  23. How to create JSON Object using String?
  24. Android how to sort JSONArray of JSONObjects
  25. how to parse a huge JSON file without loading it in memory
  26. java.lang.NoClassDefFoundError: org/apache/spark/Logging
  27. How do I use Google’s Gson API to deserialize JSON properly?
  28. Custom Jackson Serializer for a specific type in a particular class
  29. Retrieving values from nested JSON Object
  30. How to send a String array as basic name value pair as HTTPPOST?

Leave a Comment