Why does Spark RDD partition has 2GB limit for HDFS?

by

The basic abstraction for blocks in spark is a ByteBuffer, which unfortunately has a limit of Integer.MAX_VALUE (~2GB).

It is a critical issue which prevents use of spark with very large datasets.
Increasing the number of partitions can resolve it (like in OP’s case), but is not always feasible, for instance when there is large chain of transformations part of which can increase data (flatMap etc) or in cases where data is skewed.

The solution proposed is to come up with an abstraction like LargeByteBuffer which can support list of bytebuffers for a block. This impacts overall spark architecture, so it has remained unresolved for quite a while.

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. (Why) do we need to call cache or persist on a RDD
  4. Case class equality in Apache Spark
  5. Spark performance for Scala vs Python
  6. Stackoverflow due to long RDD Lineage
  7. How to find spark RDD/Dataframe size?
  8. Modify collection inside a Spark RDD foreach
  9. Parsing multiline records in Scala
  10. Explanation of fold method of spark RDD
  11. Why does partition parameter of SparkContext.textFile not take effect?
  12. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  13. How to sort an RDD in Scala Spark?
  14. How to transpose an RDD in Spark
  15. Encoder error while trying to map dataframe row to updated row
  16. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  17. How can I change column types in Spark SQL’s DataFrame?
  18. how to make saveAsTextFile NOT split output into multiple file?
  19. Spark losing println() on stdout
  20. Spark DataFrame: does groupBy after orderBy maintain that order?
  21. Spark UDAF with ArrayType as bufferSchema performance issues
  22. Apache Spark how to append new column from list/array to Spark dataframe
  23. How to get ID of a map task in Spark?
  24. Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
  25. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  26. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  27. Replace null values in Spark DataFrame
  28. Spark RDD default number of partitions
  29. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  30. How to create DataFrame from Scala’s List of Iterables?

Leave a Comment