Why does partition parameter of SparkContext.textFile not take effect?

by

If you take a look at the signature 

textFile(path: String, minPartitions: Int = defaultMinPartitions): RDD[String]

you’ll see that the argument you use is called minPartitions and this pretty much describes its function. In some cases even that is ignored but it is a different matter. Input format which is used behind the scenes still decides how to compute splits.

In this particular case you could probably use mapred.min.split.size to increase split size (this will work during load) or simply repartition after loading (this will take effect after data is loaded) but in general there should be no need for that.

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. (Why) do we need to call cache or persist on a RDD
  4. Case class equality in Apache Spark
  5. Spark performance for Scala vs Python
  6. Stackoverflow due to long RDD Lineage
  7. How to find spark RDD/Dataframe size?
  8. Modify collection inside a Spark RDD foreach
  9. Parsing multiline records in Scala
  10. Explanation of fold method of spark RDD
  11. Why does Spark RDD partition has 2GB limit for HDFS?
  12. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  13. How to sort an RDD in Scala Spark?
  14. How to transpose an RDD in Spark
  15. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  16. Spark – load CSV file as DataFrame?
  17. How to define partitioning of DataFrame?
  18. Spark dataframe write method writing many small files
  19. Filling gaps in timeseries Spark
  20. How to use COGROUP for large datasets
  21. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  22. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  23. Create new Dataframe with empty/null field values
  24. How to explode an array into multiple columns in Spark
  25. Why is join not possible after show operator?
  26. Spark Dataframe Random UUID changes after every transformation/action
  27. How to set hadoop configuration values from pyspark
  28. Spark DataFrames when udf functions do not accept large enough input variables
  29. Filter spark DataFrame on string contains
  30. Spark textFile vs wholeTextFiles

Leave a Comment