Why does Spark job fail with “too many open files”?

by

This has been answered on the spark user list:

The best way is definitely just to increase the ulimit if possible,
this is sort of an assumption we make in Spark that clusters will be
able to move it around.

You might be able to hack around this by decreasing the number of
reducers [or cores used by each node] but this could have some performance implications for your
job.

In general if a node in your cluster has C assigned cores and you run
a job with X reducers then Spark will open C*X files in parallel and
start writing. Shuffle consolidation will help decrease the total
number of files created but the number of file handles open at any
time doesn’t change so it won’t help the ulimit problem.

-Patrick Wendell

More Related Contents:

  1. How to melt Spark DataFrame?
  2. How to stop INFO messages displaying on spark console?
  3. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  4. DataFrame join optimization – Broadcast Hash Join
  5. How does Spark partition(ing) work on files in HDFS?
  6. Overwrite specific partitions in spark dataframe write method
  7. Convert date from String to Date format in Dataframes
  8. How to check if spark dataframe is empty?
  9. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  10. What is the difference between spark checkpoint and persist to a disk
  11. How to connect to remote hive server from spark [duplicate]
  12. Spark: Reading files using different delimiter than new line
  13. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  14. Adding a group count column to a PySpark dataframe
  15. Why does format(“kafka”) fail with “Failed to find data source: kafka.” (even with uber-jar)?
  16. Multiple Spark applications with HiveContext
  17. How to loop through each row of dataFrame in pyspark
  18. NoClassDefFoundError com.apache.hadoop.fs.FSDataInputStream when execute spark-shell
  19. How to group by common element in array?
  20. Spark: Best practice for retrieving big data from RDD to local machine
  21. Understanding Spark serialization
  22. Pyspark: Pass multiple columns in UDF
  23. How do I add an persistent column of row ids to Spark DataFrame?
  24. spark-shell error on Windows – can it be ignored if not using hadoop?
  25. How can I access python variable in Spark SQL?
  26. Mac spark-shell Error initializing SparkContext
  27. PySpark: How to fillna values in dataframe for specific columns?
  28. How to calculate Median in spark sqlContext for column of data type double
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Understanding Spark’s caching

Leave a Comment