Read multiline JSON in Apache Spark

by

Spark >= 2.2

Spark 2.2 introduced wholeFile multiLine option which can be used to load JSON (not JSONL) files:

spark.read
  .option("multiLine", true).option("mode", "PERMISSIVE")
  .json("/path/to/user.json")

See:

  • SPARK-18352 – Parse normal, multi-line JSON files (not just JSON Lines).
  • SPARK-20980 – Rename the option wholeFile to multiLine for JSON and CSV.

Spark < 2.2

Well, using JSONL formated data may be inconvenient but it I will argue that is not the issue with API but the format itself. JSON is simply not designed to be processed in parallel in distributed systems.

It provides no schema and without making some very specific assumptions about its formatting and shape it is almost impossible to correctly identify top level documents. Arguably this is the worst possible format to imagine to use in systems like Apache Spark. It is also quite tricky and typically impractical to write valid JSON in distributed systems.

That being said, if individual files are valid JSON documents (either single document or an array of documents) you can always try wholeTextFiles:

spark.read.json(sc.wholeTextFiles("/path/to/user.json").values())

More Related Contents:

  1. Spark Row to JSON
  2. how to convert json string to dataframe on spark
  3. Reading JSON with Apache Spark – `corrupt_record`
  4. Retain keys with null values while writing JSON in spark
  5. json.Marshal(struct) returns “{}”
  6. Why is JSON invalid if an integer begins with a leading zero?
  7. How to merge 2 JSON objects from 2 files using jq?
  8. Load images based on dynamic path in ReactJs
  9. Select objects based on value of variable in object using jq
  10. Overwrite specific partitions in spark dataframe write method
  11. How can I pass extra parameters to UDFs in Spark SQL?
  12. JSON integers: limit on size
  13. express.json vs bodyParser.json
  14. OpenAPI: what schema to accept any (complex) JSON value
  15. How to use SQL query to define table in dbtable?
  16. How do I add environment variables to launch.json in VSCode
  17. Partly JSON unmarshal into a map in Go
  18. How to create multiple tables in jasper report using json as a datasource?
  19. Cannot convert PSCustomObjects within array back to JSON correctly
  20. Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)
  21. How to use jq when the variable has reserved characters?
  22. How do I add an persistent column of row ids to Spark DataFrame?
  23. Convert null values to empty array in Spark DataFrame
  24. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  25. Check if a Postgres JSON array contains a string
  26. Calculate Cosine Similarity Spark Dataframe
  27. Filter Array Using Checkboxes with AngularJS
  28. How can I do JSON serializer ignore navigation properties?
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Error handling using JSONDecoder in Swift

Leave a Comment