What will spark do if I don’t have enough memory?

by

I think this question has been well answered in the FAQ panel of Spark website (https://spark.apache.org/faq.html):

  • What happens if my dataset does not fit in memory?
    Often each partition of data is small and does fit in memory, and these partitions are processed a few at a time. For very large partitions that do not fit in memory, Spark’s built-in operators perform external operations on datasets.
  • What happens when a cached dataset does not fit in memory?
    Spark can either spill it to disk or recompute the partitions that don’t fit in RAM each time they are requested. By default, it uses recomputation, but you can set a dataset’s storage level to MEMORY_AND_DISK to avoid this.

More Related Contents:

  1. Spark SQL: apply aggregate functions to a list of columns
  2. Unpivot in spark-sql/pyspark
  3. How to stop INFO messages displaying on spark console?
  4. DataFrame join optimization – Broadcast Hash Join
  5. Overwrite specific partitions in spark dataframe write method
  6. Convert date from String to Date format in Dataframes
  7. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  8. What is the difference between spark checkpoint and persist to a disk
  9. Xml processing in Spark
  10. What are workers, executors, cores in Spark Standalone cluster?
  11. Spark: Reading files using different delimiter than new line
  12. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  13. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  14. Why does format(“kafka”) fail with “Failed to find data source: kafka.” (even with uber-jar)?
  15. Multiple Spark applications with HiveContext
  16. How to loop through each row of dataFrame in pyspark
  17. How to group by common element in array?
  18. Understanding Spark serialization
  19. Pyspark: Pass multiple columns in UDF
  20. Spark-Obtaining file name in RDDs
  21. Convert null values to empty array in Spark DataFrame
  22. DataFrame partitionBy to a single Parquet file (per partition)
  23. Temp table caching with spark-sql
  24. spark-shell error on Windows – can it be ignored if not using hadoop?
  25. How can I access python variable in Spark SQL?
  26. How to optimize shuffle spill in Apache Spark application
  27. What is spark.driver.maxResultSize?
  28. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment