What is a task in Spark? How does the Spark worker execute the jar file?

by

When you create the SparkContext, each worker starts an executor. This is a separate process (JVM), and it loads your jar too. The executors connect back to your driver program. Now the driver can send them commands, like flatMap, map and reduceByKey in your example. When the driver quits, the executors shut down.

RDDs are sort of like big arrays that are split into partitions, and each executor can hold some of these partitions.

A task is a command sent from the driver to an executor by serializing your Function object. The executor deserializes the command (this is possible because it has loaded your jar), and executes it on a partition.

(This is a conceptual overview. I am glossing over some details, but I hope it is helpful.)

To answer your specific question: No, a new process is not started for each step. A new process is started on each worker when the SparkContext is constructed.

More Related Contents:

  1. Spark – repartition() vs coalesce()
  2. What are workers, executors, cores in Spark Standalone cluster?
  3. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  4. How does Spark aggregate function – aggregateByKey work?
  5. What is the difference between cache and persist?
  6. What is spark.driver.maxResultSize?
  7. Split Spark Dataframe string column into multiple columns
  8. What does “Stage Skipped” mean in Apache Spark web UI?
  9. How to control partition size in Spark SQL
  10. How to import multiple csv files in a single load?
  11. How are stages split into tasks in Spark?
  12. How to tune spark executor number, cores and executor memory?
  13. Which operations preserve RDD order?
  14. How to split a list to multiple columns in Pyspark?
  15. Apache Spark does not delete temporary directories
  16. How to manually set group.id and commit kafka offsets in spark structured streaming?
  17. Spark parquet partitioning : Large number of files
  18. How do you control the size of the output file?
  19. What is the maximum size for a broadcast object in Spark?
  20. Efficient pyspark join
  21. Fill in null with previously known good value with pyspark
  22. Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
  23. Keep only duplicates from a DataFrame regarding some field
  24. reading json file in pyspark
  25. Which cluster type should I choose for Spark? [closed]
  26. Overwrite only some partitions in a partitioned spark Dataset
  27. Using windowing functions in Spark
  28. PySpark: How to fillna values in dataframe for specific columns?
  29. How to calculate Median in spark sqlContext for column of data type double
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment