How to run multiple jobs in one Sparkcontext from separate threads in PySpark?

by

[*]

I was running into the same issue, so I created a tiny self-contained example. I create multiple threads using python’s threading module and submit multiple spark jobs simultaneously.

Note that by default, spark will run the jobs in First-In First-Out (FIFO): http://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application. In the example below, I change it to FAIR scheduling

# Prereqs:
# set 
# spark.dynamicAllocation.enabled         true
# spark.shuffle.service.enabled           true
  spark.scheduler.mode                    FAIR
# in spark-defaults.conf

import threading
from pyspark import SparkContext, SparkConf

def task(sc, i):
  print sc.parallelize(range(i*10000)).count()

def run_multiple_jobs():
  conf = SparkConf().setMaster('local[*]').setAppName('appname')
  # Set scheduler to FAIR: http://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application
  conf.set('spark.scheduler.mode', 'FAIR')
  sc = SparkContext(conf=conf)
  for i in range(4):
    t = threading.Thread(target=task, args=(sc, i))
    t.start()
    print 'spark task', i, 'has started'


run_multiple_jobs()

Output:

spark task 0 has started
spark task 1 has started
spark task 2 has started
spark task 3 has started
30000
0 
10000
20000

More Related Contents:

  1. How to split Vector into columns – using PySpark
  2. How to find median and quantiles using Spark
  3. Spark Dataframe distinguish columns with duplicated name
  4. How to change dataframe column names in pyspark?
  5. I can’t seem to get –py-files on Spark to work
  6. Retrieve top n in each group of a DataFrame in pyspark
  7. Pyspark: explode json in column to multiple columns
  8. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  9. Spark RDD to DataFrame python
  10. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  11. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  12. Cast column containing multiple string date formats to DateTime in Spark
  13. How to explode multiple columns of a dataframe in pyspark
  14. How to extract an element from a array in pyspark
  15. PySpark: multiple conditions in when clause
  16. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  17. Create Spark DataFrame. Can not infer schema for type
  18. Load CSV file with PySpark
  19. How to transform data with sliding window over time series data in Pyspark
  20. Create single row dataframe from list of list PySpark
  21. Build a hierarchy from a relational data-set using Pyspark
  22. Multiple condition filter on dataframe
  23. How to return a “Tuple type” in a UDF in PySpark?
  24. How to connect HBase and Spark using Python?
  25. How to process RDDs using a Python class?
  26. Pyspark 2.4.0, read avro from kafka with read stream – Python
  27. How to pivot on multiple columns in Spark SQL?
  28. Apache Spark — Assign the result of UDF to multiple dataframe columns
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment