How to calculate Median in spark sqlContext for column of data type double

by

For non integral values you should use percentile_approx UDF:

import org.apache.spark.mllib.random.RandomRDDs

val df = RandomRDDs.normalRDD(sc, 1000, 10, 1).map(Tuple1(_)).toDF("x")
df.registerTempTable("df")
sqlContext.sql("SELECT percentile_approx(x, 0.5) FROM df").show

// +--------------------+
// |                 _c0|
// +--------------------+
// |0.035379710486199915|
// +--------------------+

On a side not you should use GROUP BY not PARTITION BY. Latter one is used for window functions and has different effect than you expect.

SELECT source, percentile_approx(value, 0.5) FROM df GROUP BY source

See also How to find median using Spark

More Related Contents:

  1. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  2. How to control partition size in Spark SQL
  3. How to optimize partitioning when migrating data from JDBC source?
  4. How to connect to remote hive server from spark [duplicate]
  5. Use collect_list and collect_set in Spark SQL
  6. What is the difference between Apache Spark SQLContext vs HiveContext?
  7. Finding duplicates from large data set using Apache Spark
  8. Find maximum row per group in Spark DataFrame
  9. Split Spark Dataframe string column into multiple columns
  10. While writing to hdfs path getting error java.io.IOException: Failed to rename
  11. Multiple Aggregate operations on the same column of a spark dataframe
  12. Partitioning in spark while reading from RDBMS via JDBC
  13. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  14. How to import multiple csv files in a single load?
  15. How to access element of a VectorUDT column in a Spark DataFrame?
  16. How to save DataFrame directly to Hive?
  17. How to split a list to multiple columns in Pyspark?
  18. How to save/insert each DStream into a permanent table
  19. Spark load data and add filename as dataframe column
  20. What is the maximum size for a broadcast object in Spark?
  21. PySpark: how to resample frequencies
  22. How does createOrReplaceTempView work in Spark?
  23. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  24. pyspark: count distinct over a window
  25. Array Intersection in Spark SQL
  26. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  27. Convert between spark.SQL DataFrame and pandas DataFrame [duplicate]
  28. How to turn off scientific notation in pyspark?
  29. How to improve broadcast Join speed with between condition in Spark
  30. Spark SQL referencing attributes of UDT

Leave a Comment