What are the Spark transformations that causes a Shuffle?

by

It is actually extremely easy to find this out, without the documentation. For any of these functions just create an RDD and call to debug string, here is one example you can do the rest on ur own. 

scala> val a  = sc.parallelize(Array(1,2,3)).distinct
scala> a.toDebugString
MappedRDD[5] at distinct at <console>:12 (1 partitions)
  MapPartitionsRDD[4] at distinct at <console>:12 (1 partitions)
    **ShuffledRDD[3] at distinct at <console>:12 (1 partitions)**
      MapPartitionsRDD[2] at distinct at <console>:12 (1 partitions)
        MappedRDD[1] at distinct at <console>:12 (1 partitions)
          ParallelCollectionRDD[0] at parallelize at <console>:12 (1 partitions)

So as you can see distinct creates a shuffle. It is also particularly important to find out this way rather than docs because there are situations where a shuffle will be required or not required for a certain function. For example join usually requires a shuffle but if you join two RDD’s that branch from the same RDD spark can sometimes elide the shuffle.

More Related Contents:

  1. Spark: How to map Python with Scala or Java User Defined Functions?
  2. Resolving dependency problems in Apache Spark
  3. Add JAR files to a Spark job – spark-submit
  4. Spark read file from S3 using sc.textFile (“s3n://…)
  5. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  6. Spark Error – Unsupported class file major version
  7. Spark Strutured Streaming automatically converts timestamp to local time
  8. How can I update a broadcast variable in spark streaming?
  9. Which programming languages can I use on Android Dalvik?
  10. Apache Spark – foreach Vs foreachPartition When to use What?
  11. Matrix Multiplication in Apache Spark [closed]
  12. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  13. Is gzip format supported in Spark?
  14. Change output filename prefix for DataFrame.write()
  15. How can I force Spark to execute code?
  16. How to save models from ML Pipeline to S3 or HDFS?
  17. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  18. Running custom Java class in PySpark
  19. How to do aggregate functions, may columns and extract back
  20. Calling Java/Scala function from a task
  21. Jvm takes a long time to resolve ip-address for localhost
  22. “eval” in Scala
  23. Get the changed HTML content after it’s updated by Javascript? (htmlunit)
  24. Is there a PriorityQueue implementation with fixed capacity and custom comparator?
  25. Writing Custom Kafka Serializer
  26. Iterating over Java collections in Scala
  27. How to execute Python script from Java (via command line)?
  28. How to set timezone to UTC in Apache Spark?
  29. Why is there no GIL in the Java Virtual Machine? Why does Python need one so bad?
  30. Convert Java Map to Scala Map

Leave a Comment