Best way to get the max value in a Spark dataframe column

by 
>df1.show()
+-----+--------------------+--------+----------+-----------+
|floor|           timestamp|     uid|         x|          y|
+-----+--------------------+--------+----------+-----------+
|    1|2014-07-19T16:00:...|600dfbe2| 103.79211|71.50419418|
|    1|2014-07-19T16:00:...|5e7b40e1| 110.33613|100.6828393|
|    1|2014-07-19T16:00:...|285d22e4|110.066315|86.48873585|
|    1|2014-07-19T16:00:...|74d917a1| 103.78499|71.45633073|

>row1 = df1.agg({"x": "max"}).collect()[0]
>print row1
Row(max(x)=110.33613)
>print row1["max(x)"]
110.33613

The answer is almost the same as method3. but seems the “asDict()” in method3 can be removed

More Related Contents:

  1. Spark Equivalent of IF Then ELSE
  2. How to add a constant column in a Spark DataFrame?
  3. Load CSV file with Spark
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to perform union on two DataFrames with different amounts of columns in spark?
  6. Does spark predicate pushdown work with JDBC?
  7. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  8. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  9. Explode in PySpark
  10. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  11. Passing a data frame column and external list to udf under withColumn
  12. Updating a dataframe column in spark
  13. Dividing complex rows of dataframe to simple rows in Pyspark
  14. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  15. Spark add new column to dataframe with value from previous row
  16. PySpark create new column with mapping from a dict
  17. Pyspark: Replacing value in a column by searching a dictionary
  18. Apache Spark Python Cosine Similarity over DataFrames
  19. Pivot String column on Pyspark Dataframe
  20. GroupBy column and filter rows with maximum value in Pyspark
  21. Using UDF ignores condition in when
  22. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  23. When to cache a DataFrame?
  24. Multiple condition filter on dataframe
  25. How to return a “Tuple type” in a UDF in PySpark?
  26. How to add suffix and prefix to all columns in python/pyspark dataframe
  27. How to connect HBase and Spark using Python?
  28. Pyspark changing type of column from date to string
  29. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment