Why does sortBy transformation trigger a Spark job?

by

sortBy is implemented using sortByKey which depends on a RangePartitioner (JVM) or partitioning function (Python). When you call sortBy / sortByKey partitioner (partitioning function) is initialized eagerly and samples input RDD to compute partition boundaries. Job you see corresponds to this process.

Actual sorting is performed only if you execute an action on the newly created RDD or its descendants.

More Related Contents:

  1. Default Partitioning Scheme in Spark
  2. Unable to fetch the value of Println in apache spark
  3. Spark – repartition() vs coalesce()
  4. How does HashPartitioner work?
  5. How do I split an RDD into two or more RDDs?
  6. Partitioning in spark while reading from RDBMS via JDBC
  7. What does “Stage Skipped” mean in Apache Spark web UI?
  8. How to control partition size in Spark SQL
  9. Avoid performance impact of a single partition mode in Spark window functions
  10. How to optimize partitioning when migrating data from JDBC source?
  11. Spark: subtract two DataFrames
  12. Which operations preserve RDD order?
  13. Is groupByKey ever preferred over reduceByKey
  14. How DAG works under the covers in RDD?
  15. Spark parquet partitioning : Large number of files
  16. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  17. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  18. Apache spark dealing with case statements
  19. What is the difference between cache and persist?
  20. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  21. Spark lists all leaf node even in partitioned data
  22. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  23. Spark ALS predictAll returns empty
  24. Why is the fold action necessary in Spark?
  25. Overwrite specific partitions in spark dataframe write method
  26. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  27. How to loop through each row of dataFrame in pyspark
  28. Spark: Best practice for retrieving big data from RDD to local machine
  29. How does Spark aggregate function – aggregateByKey work?
  30. PySpark: How to fillna values in dataframe for specific columns?

Leave a Comment