Spark textFile vs wholeTextFiles

by

The main difference, as you mentioned, is that textFile will return an RDD with each line as an element while wholeTextFiles returns a PairRDD with the key being the file path. If there is no need to separate the data depending on the file, simply use textFile.

When reading uncompressed files with textFile, it will split the data into chuncks of 32MB. This is advantagous from a memory perspective. This also means that the ordering of the lines is lost, if the order should be preserved then wholeTextFiles should be used.

wholeTextFiles will read the complete content of a file at once, it won’t be partially spilled to disk or partially garbage collected. Each file will be handled by one core and the data for each file will be one a single machine making it harder to distribute the load.

More Related Contents:

  1. How to bucket the range of values from a column and count how many values fall into each interval in scala?
  2. How to store custom objects in Dataset?
  3. How do I detect if a Spark DataFrame has a column
  4. How to avoid duplicate columns after join?
  5. Flattening Rows in Spark
  6. Split 1 column into 3 columns in spark scala
  7. Provide schema while reading csv file as a dataframe
  8. How to flatmap a nested Dataframe in Spark
  9. How to force DataFrame evaluation in Spark
  10. Joining Spark dataframes on the key
  11. call of distinct and map together throws NPE in spark library
  12. Modify collection inside a Spark RDD foreach
  13. How to write unit tests in Spark 2.0+?
  14. how to filter out a null value from spark dataframe
  15. Customize SparkContext using sparkConf.set(..) when using spark-shell
  16. Replace missing values with mean – Spark Dataframe
  17. Spark: Transpose DataFrame Without Aggregating
  18. How to calculate the size of dataframe in bytes in Spark?
  19. Why does partition parameter of SparkContext.textFile not take effect?
  20. Exploding nested Struct in Spark dataframe
  21. How do I iterate RDD’s in apache spark (scala)
  22. How to compute cumulative sum using Spark
  23. Spark-Monotonically increasing id not working as expected in dataframe?
  24. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  25. Add Number of days column to Date Column in same dataframe for Spark Scala App
  26. How to sort an RDD in Scala Spark?
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment