Pyspark – converting json string to DataFrame

by

You can do the following

newJson = '{"Name":"something","Url":"https://stackoverflow.com","Author":"jangcy","BlogEntries":100,"Caller":"jangcy"}'
df = spark.read.json(sc.parallelize([newJson]))
df.show(truncate=False)

which should give 

+------+-----------+------+---------+-------------------------+
|Author|BlogEntries|Caller|Name     |Url                      |
+------+-----------+------+---------+-------------------------+
|jangcy|100        |jangcy|something|https://stackoverflow.com|
+------+-----------+------+---------+-------------------------+

More Related Contents:

  1. PySpark in iPython notebook raises Py4JJavaError when using count() and first()
  2. How to find median and quantiles using Spark
  3. Spark Dataframe distinguish columns with duplicated name
  4. How to change dataframe column names in pyspark?
  5. I can’t seem to get –py-files on Spark to work
  6. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  7. Spark RDD to DataFrame python
  8. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  9. Why is Apache-Spark – Python so slow locally as compared to pandas?
  10. Spark groupByKey alternative
  11. How to extract an element from a array in pyspark
  12. PySpark create new column with mapping from a dict
  13. Tuning parameters for implicit pyspark.ml ALS matrix factorization model through pyspark.ml CrossValidator
  14. Spark iteration time increasing exponentially when using join
  15. Create Spark DataFrame. Can not infer schema for type
  16. PySpark DataFrames – way to enumerate without converting to Pandas?
  17. Getting Spark, Python, and MongoDB to work together
  18. How to transform data with sliding window over time series data in Pyspark
  19. Using UDF ignores condition in when
  20. Create single row dataframe from list of list PySpark
  21. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  22. pyspark parse fixed width text file
  23. Concatenating string by rows in pyspark
  24. What is the equivalent to scala.util.Try in pyspark?
  25. Concatenate two PySpark dataframes
  26. How do I convert an array (i.e. list) column to Vector
  27. Filtering DataFrame using the length of a column
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe
  30. Random numbers generation in PySpark

Leave a Comment