Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames

by

Since Spark 1.6 you can use pivot function on GroupedData and provide aggregate expression.

pivoted = (df
    .groupBy("ID", "Age")
    .pivot(
        "Country",
        ['US', 'UK', 'CA'])  # Optional list of levels
    .sum("Score"))  # alternatively you can use .agg(expr))
pivoted.show()

## +---+---+---+---+---+
## | ID|Age| US| UK| CA|
## +---+---+---+---+---+
## |X01| 41|  3|  1|  2|
## |X02| 72|  4|  6|  7|
## +---+---+---+---+---+

Levels can be omitted but if provided can both boost performance and serve as an internal filter.

This method is still relatively slow but certainly beats manual passing data manually between JVM and Python.

More Related Contents:

  1. How to pivot on multiple columns in Spark SQL?
  2. Convert pyspark string to date format
  3. How to split Vector into columns – using PySpark
  4. How to use JDBC source to write and read data in (Py)Spark?
  5. Spark Dataframe distinguish columns with duplicated name
  6. Pyspark: Split multiple array columns into rows
  7. How to change dataframe column names in pyspark?
  8. Retrieve top n in each group of a DataFrame in pyspark
  9. Pyspark: explode json in column to multiple columns
  10. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  11. How to use a Scala class inside Pyspark
  12. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  13. Why is Apache-Spark – Python so slow locally as compared to pandas?
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. PySpark: multiple conditions in when clause
  17. PySpark create new column with mapping from a dict
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Create Spark DataFrame. Can not infer schema for type
  20. Load CSV file with PySpark
  21. Create single row dataframe from list of list PySpark
  22. Cannot find col function in pyspark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. How to pass a constant value to Python UDF?
  25. How to zip two array columns in Spark SQL
  26. When to cache a DataFrame?
  27. Multiple condition filter on dataframe
  28. How to return a “Tuple type” in a UDF in PySpark?
  29. How to connect HBase and Spark using Python?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment