get TopN of all groups after group by using Spark DataFrame

by

You can use rank window function as follows

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.{rank, desc}

val n: Int = ???

// Window definition
val w = Window.partitionBy($"user").orderBy(desc("rating"))

// Filter
df.withColumn("rank", rank.over(w)).where($"rank" <= n)

If you don’t care about ties then you can replace rank with row_number

More Related Contents:

  1. How to select the first row of each group?
  2. Querying Spark SQL DataFrame with complex types
  3. Including null values in an Apache Spark Join
  4. Difference between === null and isNull in Spark DataDrame
  5. Concatenate columns in Apache Spark DataFrame
  6. Spark Window Functions – rangeBetween dates
  7. Spark SQL window function with complex condition
  8. Does SparkSQL support subquery?
  9. Spark sql queries vs dataframe functions
  10. Explode (transpose?) multiple columns in Spark SQL table
  11. How to get other columns when using Spark DataFrame groupby?
  12. Why using a UDF in a SQL query leads to cartesian product?
  13. How to aggregate over rolling time window with groups in Spark
  14. What is the difference between cube, rollup and groupBy operators?
  15. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  16. How to define schema for custom type in Spark SQL?
  17. Spark unionAll multiple dataframes
  18. How can I pass extra parameters to UDFs in Spark SQL?
  19. How to split a dataframe into dataframes with same column values?
  20. How to create an empty DataFrame with a specified schema?
  21. How to define a custom aggregation function to sum a column of Vectors?
  22. Spark DataFrame: does groupBy after orderBy maintain that order?
  23. How to construct Dataframe from a Excel (xls,xlsx) file in Scala Spark?
  24. Better way to convert a string field into timestamp in Spark
  25. Spark UDAF with ArrayType as bufferSchema performance issues
  26. Create new Dataframe with empty/null field values
  27. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  28. Calculate Cosine Similarity Spark Dataframe
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment