How can I force Spark to execute code?

by

Short answer:

To force Spark to execute a transformation, you’ll need to require a result. Sometimes a simple count action is sufficient.

TL;DR:

Ok, let’s review the RDD operations.

RDDs support two types of operations:

  • transformations – which create a new dataset from an existing one.
  • actions – which return a value to the driver program after running a computation on the dataset.

For example, map is a transformation that passes each dataset element through a function and returns a new RDD representing the results. On the other hand, reduce is an action that aggregates all the elements of the RDD using some function and returns the final result to the driver program (although there is also a parallel reduceByKey that returns a distributed dataset).

All transformations in Spark are lazy, in that they do not compute their results right away.

Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program. This design enables Spark to run more efficiently – for example, we can realize that a dataset created through map will be used in a reduce and return only the result of the reduce to the driver, rather than the larger mapped dataset.

By default, each transformed RDD may be recomputed each time you run an action on it. However, you may also persist an RDD in memory using the persist (or cache) method, in which case Spark will keep the elements around on the cluster for much faster access the next time you query it. There is also support for persisting RDDs on disk, or replicated across multiple nodes.

Conclusion

To force Spark to execute a call to map, you’ll need to require a result. Sometimes a count action is sufficient.

Reference

More Related Contents:

  1. Resolving dependency problems in Apache Spark
  2. Add JAR files to a Spark job – spark-submit
  3. Spark: How to map Python with Scala or Java User Defined Functions?
  4. Read whole text files from a compression in Spark
  5. Spark read file from S3 using sc.textFile (“s3n://…)
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. How can I update a broadcast variable in spark streaming?
  8. Apache Spark – foreach Vs foreachPartition When to use What?
  9. Matrix Multiplication in Apache Spark [closed]
  10. What are the Spark transformations that causes a Shuffle?
  11. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  12. Is gzip format supported in Spark?
  13. Change output filename prefix for DataFrame.write()
  14. How to save models from ML Pipeline to S3 or HDFS?
  15. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  16. Spark – load CSV file as DataFrame?
  17. How to use java.String.format in Scala?
  18. Scala equivalent of Java java.lang.Class Object
  19. Calling a mapreduce job from a simple java program
  20. Shuffle a list of integers with Java 8 Streams API
  21. Sign CSR using Bouncy Castle
  22. Hadoop DistributedCache is deprecated – what is the preferred API?
  23. Java compile speed vs Scala compile speed
  24. Iterate twice on values (MapReduce)
  25. NotSerializableException on anonymous class
  26. How can I use a Scala singleton object in Java?
  27. How do I call a UDF on a Spark DataFrame using JAVA?
  28. How to get element by Index in Spark RDD (Java)
  29. How to make Eclipse see the changes in Play! compiled templates?
  30. Running custom Java class in PySpark

Leave a Comment