How jobs are assigned to executors in Spark Streaming?

by

Actually, in the current implementation of Spark Streaming and under default configuration, only job is active (i.e. under execution) at any point of time. So if one batch’s processing takes longer than 10 seconds, then then next batch’s jobs will stay queued.

This can be changed with an experimental Spark property “spark.streaming.concurrentJobs” which is by default set to 1. Its not currently documented (maybe I should add it).

The reason it is set to 1 is that concurrent jobs can potentially lead to weird sharing of resources and which can make it hard to debug the whether there is sufficient resources in the system to process the ingested data fast enough. With only 1 job running at a time, it is easy to see that if batch processing time < batch interval, then the system will be stable. Granted that this may not be the most efficient use of resources under certain conditions. We definitely hope to improve this in the future.

There is a little bit of material regarding the internals of Spark Streaming in this meetup slides (sorry, about the shameless self advertising 🙂 ). That may be useful to you.

More Related Contents:

  1. How to split Vector into columns – using PySpark
  2. Split Spark Dataframe string column into multiple columns
  3. What does “Stage Skipped” mean in Apache Spark web UI?
  4. How to control partition size in Spark SQL
  5. Split 1 column into 3 columns in spark scala
  6. Pyspark: explode json in column to multiple columns
  7. How to split a list to multiple columns in Pyspark?
  8. Matrix Multiplication in Apache Spark [closed]
  9. Cast column containing multiple string date formats to DateTime in Spark
  10. How to manually set group.id and commit kafka offsets in spark structured streaming?
  11. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  12. Spark Row to JSON
  13. Easiest way to install Python dependencies on Spark executor nodes?
  14. How do I iterate RDD’s in apache spark (scala)
  15. Spark-Monotonically increasing id not working as expected in dataframe?
  16. reduce result datasets into single dataset
  17. How to print the contents of RDD?
  18. How to exclude multiple columns in Spark dataframe in Python
  19. Why does Spark job fail with “too many open files”?
  20. How does Distinct() function work in Spark?
  21. How to read input from S3 in a Spark Streaming EC2 cluster application
  22. Stratified sampling with pyspark
  23. Get CSV to Spark dataframe
  24. DBMS_JOB vs DBMS_SCHEDULER
  25. How to process RDDs using a Python class?
  26. Scala-Spark Dynamically call groupby and agg with parameter values
  27. Joining two dataframes without a common column
  28. What is the Spark DataFrame method `toPandas` actually doing?
  29. how to add Row id in pySpark dataframes [duplicate]
  30. Difference between === null and isNull in Spark DataDrame

Leave a Comment