How does Spark partition(ing) work on files in HDFS?

by

When Spark reads a file from HDFS, it creates a single partition for a single input split. Input split is set by the Hadoop InputFormat used to read this file. For instance, if you use textFile() it would be TextInputFormat in Hadoop, which would return you a single partition for a single block of HDFS (but the split between partitions would be done on line split, not the exact block split), unless you have a compressed text file. In case of compressed file you would get a single partition for a single file (as compressed text files are not splittable).

When you call rdd.repartition(x) it would perform a shuffle of the data from N partititons you have in rdd to x partitions you want to have, partitioning would be done on round robin basis.

If you have a 30GB uncompressed text file stored on HDFS, then with the default HDFS block size setting (128MB) it would be stored in 235 blocks, which means that the RDD you read from this file would have 235 partitions. When you call repartition(1000) your RDD would be marked as to be repartitioned, but in fact it would be shuffled to 1000 partitions only when you will execute an action on top of this RDD (lazy execution concept)

More Related Contents:

  1. While writing to hdfs path getting error java.io.IOException: Failed to rename
  2. Find maximum row per group in Spark DataFrame
  3. How to control partition size in Spark SQL
  4. How to handle categorical features with spark-ml?
  5. How to access element of a VectorUDT column in a Spark DataFrame?
  6. How are stages split into tasks in Spark?
  7. How to tune spark executor number, cores and executor memory?
  8. Python Spark Cumulative Sum by Group Using DataFrame
  9. Link Spark with iPython Notebook
  10. Default Partitioning Scheme in Spark
  11. How to manually set group.id and commit kafka offsets in spark structured streaming?
  12. Spark parquet partitioning : Large number of files
  13. How do you control the size of the output file?
  14. Any performance issues forcing eager evaluation using count in spark?
  15. What is a task in Spark? How does the Spark worker execute the jar file?
  16. Debugging “Managed memory leak detected” in Spark 1.6.0
  17. Spark: how to get the number of written rows?
  18. Why are Spark Parquet files for an aggregate larger than the original?
  19. Apache spark dealing with case statements
  20. What’s the difference between Spark ML and MLLIB packages
  21. Spark iterate HDFS directory
  22. pyspark: count distinct over a window
  23. Mind blown: RDD.zip() method
  24. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  25. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  26. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  27. PySpark: How to fillna values in dataframe for specific columns?
  28. How to calculate Median in spark sqlContext for column of data type double
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Understanding Spark’s caching

Leave a Comment