How to sort an RDD in Scala Spark?

by

If you only need the top 10, use rdd.top(10). It avoids sorting, so it is faster.

rdd.top makes one parallel pass through the data, collecting the top N in each partition in a heap, then merges the heaps. It is an O(rdd.count) operation. Sorting would be O(rdd.count log rdd.count), and incur a lot of data transfer — it does a shuffle, so all of the data would be transmitted over the network.

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. (Why) do we need to call cache or persist on a RDD
  4. Case class equality in Apache Spark
  5. Spark performance for Scala vs Python
  6. Stackoverflow due to long RDD Lineage
  7. How to find spark RDD/Dataframe size?
  8. Modify collection inside a Spark RDD foreach
  9. Parsing multiline records in Scala
  10. Explanation of fold method of spark RDD
  11. Why does partition parameter of SparkContext.textFile not take effect?
  12. Why does Spark RDD partition has 2GB limit for HDFS?
  13. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  14. How to transpose an RDD in Spark
  15. Automatically and Elegantly flatten DataFrame in Spark SQL
  16. How do I skip a header from CSV files in Spark?
  17. Spark extracting values from a Row
  18. How to pass -D parameter or environment variable to Spark job?
  19. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  20. Append a column to Data Frame in Apache Spark 1.3
  21. Processing multiple files as independent RDD’s in parallel
  22. MatchError while accessing vector column in Spark 2.0
  23. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  24. Spark / Scala: forward fill with last observation
  25. How to convert unix timestamp to date in Spark
  26. Apache Spark: Get number of records per partition
  27. Derive multiple columns from a single column in a Spark DataFrame
  28. How to save a spark DataFrame as csv on disk?
  29. How to process multi line input records in Spark
  30. How to sort by column in descending order in Spark SQL?

Leave a Comment