how to convert json string to dataframe on spark

by

For Spark 2.2+:

import spark.implicits._
val jsonStr = """{ "metadata": { "key": 84896, "value": 54 }}"""
val df = spark.read.json(Seq(jsonStr).toDS)

For Spark 2.1.x:

val events = sc.parallelize("""{"action":"create","timestamp":"2016-01-07T00:01:17Z"}""" :: Nil)    
val df = sqlContext.read.json(events)

Hint: this is using sqlContext.read.json(jsonRDD: RDD[Stirng]) overload.
There is also sqlContext.read.json(path: String) where it reads a Json file directly.

For older versions:

val jsonStr = """{ "metadata": { "key": 84896, "value": 54 }}"""
val rdd = sc.parallelize(Seq(jsonStr))
val df = sqlContext.read.json(rdd)

More Related Contents:

  1. Spark Row to JSON
  2. Reading JSON with Apache Spark – `corrupt_record`
  3. How to query JSON data column using Spark DataFrames?
  4. What JSON library to use in Scala? [closed]
  5. How to define partitioning of DataFrame?
  6. How to parse JSON in Scala using standard Scala classes?
  7. How do I detect if a Spark DataFrame has a column
  8. How to split a dataframe into dataframes with same column values?
  9. How to create an empty DataFrame with a specified schema?
  10. Provide schema while reading csv file as a dataframe
  11. Dropping a nested column from Spark DataFrame
  12. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  13. How to zip two (or more) DataFrame in Spark
  14. Convert Pandas DataFrame to JSON format
  15. Joining Spark dataframes on the key
  16. Apache Spark how to append new column from list/array to Spark dataframe
  17. Defaults for missing properties in play 2 JSON formats
  18. How to turn json to case class when case class has only one field
  19. Spark Dataframes UPSERT to Postgres Table
  20. No Json serializer as JsObject found for type play.api.libs.json.JsObject
  21. Create new Dataframe with empty/null field values
  22. Provide schema while reading csv file as a dataframe in Scala Spark
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. Replace missing values with mean – Spark Dataframe
  25. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  26. Replace null values in Spark DataFrame
  27. How to show full column content in a Spark Dataframe?
  28. Spark Dataframe Random UUID changes after every transformation/action
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment