TaskSchedulerImpl: Initial job has not accepted any resources;

by

I faced similar issue and after some online research and trial-n-error, I narrowed down to 3 causes for this (except for the first the other two are not even close to the error message):

  1. As indicated by the error, probably you are allocating the resources more than that is available. => This was not my issue
  2. Hostname & IP Address mishaps: I took care of this by specifying the SPARK_MASTER_IP and SPARK_LOCAL_IP in spark-env.sh
  3. Disable Firewall on the client : This was the solution that worked for me. Since I was working on a prototype in-house code, I disabled the firewall on the client node. For some reason the worker nodes, were not able to talk back to the client for me. For production purposes, you would want to open-up certain number of ports required.

More Related Contents:

  1. java.lang.NoClassDefFoundError: org/apache/spark/Logging
  2. com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces
  3. How to handle this scenario in spark? [closed]
  4. Add JAR files to a Spark job – spark-submit
  5. How to flatten a struct in a Spark dataframe?
  6. Spark: How to map Python with Scala or Java User Defined Functions?
  7. Read whole text files from a compression in Spark
  8. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  9. Spark read file from S3 using sc.textFile (“s3n://…)
  10. java.lang.ClassCastException using lambda expressions in spark job on remote server
  11. Spark Strutured Streaming automatically converts timestamp to local time
  12. How can I update a broadcast variable in spark streaming?
  13. Is the IN relation in Cassandra bad for queries?
  14. Matrix Multiplication in Apache Spark [closed]
  15. spark broadcast variable Map giving null value
  16. What are the Spark transformations that causes a Shuffle?
  17. Cassandra – Is there a way to limit number of async queries?
  18. Is there a reason not to use SparkContext.getOrCreate when writing a spark job?
  19. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  20. Is gzip format supported in Spark?
  21. Change output filename prefix for DataFrame.write()
  22. How can I force Spark to execute code?
  23. How do I call a UDF on a Spark DataFrame using JAVA?
  24. How to get element by Index in Spark RDD (Java)
  25. Retain keys with null values while writing JSON in spark
  26. How to save models from ML Pipeline to S3 or HDFS?
  27. Reading data from Azure Blob with Spark
  28. Converting mysql table to spark dataset is very slow compared to same from csv file
  29. How to use Column.isin in Java?
  30. Running custom Java class in PySpark

Leave a Comment