How to do aggregate functions, may columns and extract back

by

If you are going to make all the groupBy with the same columns, and you want to create all the aggregation column references as avg(col_interested ).as(col_interested_avg) with all the elements of columns_interestedList you can create all the references with the stream, and pass them to the gag method.

List<Column> avgCols = columns_interestedList.stream()
.map(col_interested -> avg(col_interested).as(col_interested + "_avg"))
.collect(Collectors.toList());

dataDs.groupBy("grp_col_1","grp_col_1")
.agg(avgCols.add(count((*).as("col_interested_count").toArray(new Column[0]));

More Related Contents:

  1. How to handle this scenario in spark? [closed]
  2. How to flatten a struct in a Spark dataframe?
  3. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  4. Spark sql how to explode without losing null values
  5. Spark Strutured Streaming automatically converts timestamp to local time
  6. spark broadcast variable Map giving null value
  7. How to set timezone to UTC in Apache Spark?
  8. Change output filename prefix for DataFrame.write()
  9. How do I call a UDF on a Spark DataFrame using JAVA?
  10. Retain keys with null values while writing JSON in spark
  11. How to use Column.isin in Java?
  12. Why does SparkSession execute twice for one action?
  13. Resolving dependency problems in Apache Spark
  14. Read whole text files from a compression in Spark
  15. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  16. java.lang.ClassCastException using lambda expressions in spark job on remote server
  17. Spark Error – Unsupported class file major version
  18. Apache Spark – foreach Vs foreachPartition When to use What?
  19. Timestamp convert [duplicate]
  20. What are the Spark transformations that causes a Shuffle?
  21. PySpark: java.lang.OutofMemoryError: Java heap space
  22. TaskSchedulerImpl: Initial job has not accepted any resources;
  23. java.lang.NoClassDefFoundError: org/apache/spark/Logging
  24. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  25. How can I force Spark to execute code?
  26. How to get element by Index in Spark RDD (Java)
  27. Reading data from Azure Blob with Spark
  28. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  29. Spark: get number of cluster cores programmatically
  30. Running custom Java class in PySpark

Leave a Comment