What will spark do if I don’t have enough memory?
I think this question has been well answered in the FAQ panel of Spark website (https://spark.apache.org/faq.html): What happens if my dataset does not fit in memory? Often each partition of data is small and does fit in memory, and these partitions are processed a few at a time. For very large partitions that do not … Read more