Which operations preserve RDD order?

by

All operations preserve the order, except those that explicitly do not. Ordering is always “meaningful”, not just after a sortBy. For example, if you read a file (sc.textFile) the lines of the RDD will be in the order that they were in the file.

Without trying to give a complete list, map, filter and flatMap do preserve the order. sortBy, partitionBy, join do not preserve the order.

The reason is that most RDD operations work on Iterators inside the partitions. So map or filter just has no way to mess up the order. You can take a look at the code to see for yourself.

You may now ask: What if I have an RDD with a HashPartitioner. What happens when I use map to change the keys? Well, they will stay in place, and now the RDD is not partitioned by the key. You can use partitionBy to restore the partitioning with a shuffle.

More Related Contents:

  1. Unable to fetch the value of Println in apache spark
  2. Spark – repartition() vs coalesce()
  3. How do I split an RDD into two or more RDDs?
  4. What does “Stage Skipped” mean in Apache Spark web UI?
  5. Spark: subtract two DataFrames
  6. Is groupByKey ever preferred over reduceByKey
  7. Default Partitioning Scheme in Spark
  8. How DAG works under the covers in RDD?
  9. Spark parquet partitioning : Large number of files
  10. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  11. Why does sortBy transformation trigger a Spark job?
  12. Apache spark dealing with case statements
  13. What is the difference between cache and persist?
  14. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  15. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  16. Spark ALS predictAll returns empty
  17. Why is the fold action necessary in Spark?
  18. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  19. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  20. Spark SQL – load data with JDBC using SQL statement, not table name
  21. How to optimize partitioning when migrating data from JDBC source?
  22. How to improve performance for slow Spark jobs using DataFrame and JDBC connection?
  23. Best Practice to launch Spark Applications via Web Application?
  24. Spark yarn cluster vs client – how to choose which one to use?
  25. Spark on YARN + Secured hbase
  26. Spark load data and add filename as dataframe column
  27. Spark groupByKey alternative
  28. Explode array data into rows in spark [duplicate]
  29. PySpark DataFrames – way to enumerate without converting to Pandas?
  30. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?

Leave a Comment