Read XML in spark

by

heirarchy should be rootTag and att should be rowTag as 

df = spark.read \
    .format("com.databricks.spark.xml") \
    .option("rootTag", "hierarchy") \
    .option("rowTag", "att") \
    .load("test.xml")

and you should get 

+-----+------+----------------------------+
|Order|attval|children                    |
+-----+------+----------------------------+
|1    |Data  |[[[1, Studyval], [2, Site]]]|
|2    |Info  |[[[1, age], [2, gender]]]   |
+-----+------+----------------------------+

and schema

root
 |-- Order: long (nullable = true)
 |-- attval: string (nullable = true)
 |-- children: struct (nullable = true)
 |    |-- att: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- Order: long (nullable = true)
 |    |    |    |-- attval: string (nullable = true)

find more information on databricks xml

More Related Contents:

  1. How to make good reproducible Apache Spark examples
  2. How to pivot Spark DataFrame?
  3. How to add a constant column in a Spark DataFrame?
  4. Spark Dataframe distinguish columns with duplicated name
  5. How do I add a new column to a Spark DataFrame (using PySpark)?
  6. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  7. How to access element of a VectorUDT column in a Spark DataFrame?
  8. Retrieve top n in each group of a DataFrame in pyspark
  9. Rename nested field in spark dataframe
  10. Filtering a Pyspark DataFrame with SQL-like IN clause
  11. Filter Spark DataFrame based on another DataFrame that specifies denylist criteria
  12. Updating a dataframe column in spark
  13. Dividing complex rows of dataframe to simple rows in Pyspark
  14. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  15. R: convert XML data to data frame
  16. How to explode multiple columns of a dataframe in pyspark
  17. Spark add new column to dataframe with value from previous row
  18. How to loop through each row of dataFrame in pyspark
  19. Why does Spark think this is a cross / Cartesian join
  20. Pyspark: Replacing value in a column by searching a dictionary
  21. Create Spark DataFrame. Can not infer schema for type
  22. Pivot String column on Pyspark Dataframe
  23. How to transform XML data into a data.frame?
  24. Pyspark filter dataframe by columns of another dataframe
  25. spark dataframe drop duplicates and keep first
  26. Filtering DataFrame using the length of a column
  27. How to exclude multiple columns in Spark dataframe in Python
  28. Multiple condition filter on dataframe
  29. How to return a “Tuple type” in a UDF in PySpark?
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment