How spark read a large file (petabyte) when file can not be fit in spark’s main memory

by

First of all, Spark only starts reading in the data when an action (like count, collect or write) is called. Once an action is called, Spark loads in data in partitions – the number of concurrently loaded partitions depend on the number of cores you have available. So in Spark you can think of 1 partition = 1 core = 1 task. Note that all concurrently loaded partitions have to fit into memory, or you will get an OOM.

Assuming that you have several stages, Spark then runs the transformations from the first stage on the loaded partitions only. Once it has applied the transformations on the data in the loaded partitions, it stores the output as shuffle-data and then reads in more partitions. It then applies the transformations on these partitions, stores the output as shuffle-data, reads in more partitions and so forth until all data has been read.

If you apply no transformation but only do for instance a count, Spark will still read in the data in partitions, but it will not store any data in your cluster and if you do the count again it will read in all the data once again. To avoid reading in data several times, you might call cache or persist in which case Spark will try to store the data in you cluster. On cache (which is the same as persist(StorageLevel.MEMORY_ONLY) it will store all partitions in memory – if it doesn’t fit in memory you will get an OOM. If you call persist(StorageLevel.MEMORY_AND_DISK) it will store as much as it can in memory and the rest will be put on disk. If data doesn’t fit on disk either the OS will usually kill your workers.

Note that Spark has its own little memory management system. Some of the memory that you assign to your Spark job is used to hold the data being worked on and some of the memory is used for storage if you call cache or persist.

I hope this explanation helps 🙂

More Related Contents:

  1. Unable to fetch the value of Println in apache spark
  2. Spark – repartition() vs coalesce()
  3. How do I split an RDD into two or more RDDs?
  4. What does “Stage Skipped” mean in Apache Spark web UI?
  5. Spark: subtract two DataFrames
  6. Which operations preserve RDD order?
  7. Is groupByKey ever preferred over reduceByKey
  8. Default Partitioning Scheme in Spark
  9. How DAG works under the covers in RDD?
  10. Spark parquet partitioning : Large number of files
  11. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  12. Why does sortBy transformation trigger a Spark job?
  13. Apache spark dealing with case statements
  14. What is the difference between cache and persist?
  15. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  16. Spark ALS predictAll returns empty
  17. Why is the fold action necessary in Spark?
  18. How to find median and quantiles using Spark
  19. (Why) do we need to call cache or persist on a RDD
  20. What is the difference between map and flatMap and a good use case for each?
  21. How to save/insert each DStream into a permanent table
  22. Spark yarn cluster vs client – how to choose which one to use?
  23. Spark on YARN + Secured hbase
  24. Spark groupByKey alternative
  25. PySpark: how to resample frequencies
  26. Explode array data into rows in spark [duplicate]
  27. PySpark DataFrames – way to enumerate without converting to Pandas?
  28. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  29. When are accumulators truly reliable?
  30. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:

Leave a Comment