how to make saveAsTextFile NOT split output into multiple file?

by

The reason it saves it as multiple files is because the computation is distributed. If the output is small enough such that you think you can fit it on one machine, then you can end your program with 

val arr = year.collect()

And then save the resulting array as a file, Another way would be to use a custom partitioner, partitionBy, and make it so everything goes to one partition though that isn’t advisable because you won’t get any parallelization.

If you require the file to be saved with saveAsTextFile you can use coalesce(1,true).saveAsTextFile(). This basically means do the computation then coalesce to 1 partition. You can also use repartition(1) which is just a wrapper for coalesce with the shuffle argument set to true. Looking through the source of RDD.scala is how I figured most of this stuff out, you should take a look.

More Related Contents:

  1. How to bucket the range of values from a column and count how many values fall into each interval in scala?
  2. How to store custom objects in Dataset?
  3. How to query JSON data column using Spark DataFrames?
  4. How do I detect if a Spark DataFrame has a column
  5. Split 1 column into 3 columns in spark scala
  6. Spark unionAll multiple dataframes
  7. How can I pass extra parameters to UDFs in Spark SQL?
  8. Spark performance for Scala vs Python
  9. How to zip two (or more) DataFrame in Spark
  10. How to force DataFrame evaluation in Spark
  11. How to write spark streaming DF to Kafka topic
  12. Stackoverflow due to long RDD Lineage
  13. call of distinct and map together throws NPE in spark library
  14. How to get path to the uploaded file
  15. how to filter out a null value from spark dataframe
  16. Is there a reason not to use SparkContext.getOrCreate when writing a spark job?
  17. What are the various join types in Spark?
  18. Provide schema while reading csv file as a dataframe in Scala Spark
  19. Replace missing values with mean – Spark Dataframe
  20. How to calculate the size of dataframe in bytes in Spark?
  21. Why does partition parameter of SparkContext.textFile not take effect?
  22. Spark Scala list folders in directory
  23. Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)
  24. Spark-Monotonically increasing id not working as expected in dataframe?
  25. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  26. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  27. Spark : how to run spark file from spark shell
  28. How can I connect to a postgreSQL database into Apache Spark using scala?
  29. Spark column string replace when present in other column (row)
  30. How to vectorize DataFrame columns for ML algorithms?

Leave a Comment