reading json file in pyspark

by

First of all, the json is invalid. After the header a , is missing.

That being said, lets take this json:

{"header":{"platform":"atm","version":"2.0"},"details":[{"abc":"3","def":"4"},{"abc":"5","def":"6"},{"abc":"7","def":"8"}]}

This can be processed by:

>>> df = sqlContext.jsonFile('test.json')
>>> df.first()
Row(details=[Row(abc="3", def="4"), Row(abc="5", def="6"), Row(abc="7", def="8")], header=Row(platform='atm', version='2.0'))

>>> df = df.flatMap(lambda row: row['details'])
PythonRDD[38] at RDD at PythonRDD.scala:43

>>> df.collect()
[Row(abc="3", def="4"), Row(abc="5", def="6"), Row(abc="7", def="8")]

>>> df.map(lambda entry: (int(entry['abc']),     int(entry['def']))).collect()
[(3, 4), (5, 6), (7, 8)]

Hope this helps!

More Related Contents:

  1. Spark Dataframe validating column names for parquet writes
  2. How to save/insert each DStream into a permanent table
  3. Finding duplicates from large data set using Apache Spark
  4. Using a column value as a parameter to a spark DataFrame function
  5. How do I split an RDD into two or more RDDs?
  6. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  7. Split Spark Dataframe string column into multiple columns
  8. Avoid performance impact of a single partition mode in Spark window functions
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. Python Spark Cumulative Sum by Group Using DataFrame
  11. How to split a list to multiple columns in Pyspark?
  12. pyspark dataframe filter or include based on list
  13. Link Spark with iPython Notebook
  14. Efficient pyspark join
  15. How to loop through each row of dataFrame in pyspark
  16. Fill in null with previously known good value with pyspark
  17. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  18. PySpark – get row number for each row in a group
  19. Save ML model for future usage
  20. Apache spark dealing with case statements
  21. Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
  22. Keep only duplicates from a DataFrame regarding some field
  23. How can I access python variable in Spark SQL?
  24. pyspark: rolling average using timeseries data
  25. How to exclude multiple columns in Spark dataframe in Python
  26. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  27. Stratified sampling with pyspark
  28. Get CSV to Spark dataframe
  29. How to turn off scientific notation in pyspark?
  30. Spark ALS predictAll returns empty

Leave a Comment