Pyspark: Parse a column of json strings

by

For Spark 2.1+, you can use from_json which allows the preservation of the other non-json columns within the dataframe as follows:

from pyspark.sql.functions import from_json, col
json_schema = spark.read.json(df.rdd.map(lambda row: row.json)).schema
df.withColumn('json', from_json(col('json'), json_schema))

You let Spark derive the schema of the json string column. Then the df.json column is no longer a StringType, but the correctly decoded json structure, i.e., nested StrucType and all the other columns of df are preserved as-is.

You can access the json content as follows:

df.select(col('json.header').alias('header'))

More Related Contents:

  1. How to split Vector into columns – using PySpark
  2. How to find median and quantiles using Spark
  3. Spark Dataframe distinguish columns with duplicated name
  4. How to change dataframe column names in pyspark?
  5. I can’t seem to get –py-files on Spark to work
  6. Retrieve top n in each group of a DataFrame in pyspark
  7. Pyspark: explode json in column to multiple columns
  8. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  9. Spark RDD to DataFrame python
  10. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  11. Why is Apache-Spark – Python so slow locally as compared to pandas?
  12. Cast column containing multiple string date formats to DateTime in Spark
  13. Spark groupByKey alternative
  14. How to extract an element from a array in pyspark
  15. PySpark create new column with mapping from a dict
  16. Tuning parameters for implicit pyspark.ml ALS matrix factorization model through pyspark.ml CrossValidator
  17. Spark iteration time increasing exponentially when using join
  18. Create Spark DataFrame. Can not infer schema for type
  19. PySpark DataFrames – way to enumerate without converting to Pandas?
  20. How to transform data with sliding window over time series data in Pyspark
  21. Create single row dataframe from list of list PySpark
  22. _corrupt_record error when reading a JSON file into Spark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. Build a hierarchy from a relational data-set using Pyspark
  25. Concatenate two PySpark dataframes
  26. How do I convert an array (i.e. list) column to Vector
  27. Filtering DataFrame using the length of a column
  28. How to process RDDs using a Python class?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment