Spark: long delay between jobs

by

I/O operations often come with significant overhead that will occur on the master node. Since this work isn’t parallelized, it can take quite a bit of time. And since it is not a job, it does not show up in the resource manager UI. Some examples of I/O tasks that are done by the master node

  • Spark will write to temporary s3 directories, then move the files using the master node
  • Reading of text files often occur on the master node
  • When writing parquet files, the master node will scan all the files post-write to check the schema

These issues can be solved by tweaking yarn settings or redesigning your code. If you provide some source code, I might be able to pinpoint your issue.

Discussion of writing I/O Overhead with Parquet and s3

Discussion of reading I/O Overhead “s3 is not a filesystem”

More Related Contents:

  1. Write to multiple outputs by key Spark – one Spark job
  2. Spark – load CSV file as DataFrame?
  3. How to get ID of a map task in Spark?
  4. Apache Spark: Get number of records per partition
  5. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  6. Spark Scala list folders in directory
  7. How to define schema for custom type in Spark SQL?
  8. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  9. NullPointerException in Scala Spark, appears to be caused be collection type?
  10. What does setMaster `local[*]` mean in spark?
  11. Spark unionAll multiple dataframes
  12. How can I pass extra parameters to UDFs in Spark SQL?
  13. How to split a dataframe into dataframes with same column values?
  14. Spark performance for Scala vs Python
  15. How to use regex to include/exclude some input files in sc.textFile?
  16. How to define a custom aggregation function to sum a column of Vectors?
  17. Spark losing println() on stdout
  18. How to write spark streaming DF to Kafka topic
  19. Stackoverflow due to long RDD Lineage
  20. Better way to convert a string field into timestamp in Spark
  21. Spark UDAF with ArrayType as bufferSchema performance issues
  22. How to find spark RDD/Dataframe size?
  23. Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
  24. Provide schema while reading csv file as a dataframe in Scala Spark
  25. Spark RDD default number of partitions
  26. Why does Spark RDD partition has 2GB limit for HDFS?
  27. Calculate Cosine Similarity Spark Dataframe
  28. How to print the contents of RDD?
  29. Scala-Spark Dynamically call groupby and agg with parameter values
  30. Joining two dataframes without a common column

Leave a Comment