Write single CSV file using spark-csv

by

It is creating a folder with multiple files, because each partition is saved individually. If you need a single output file (still in a folder) you can repartition (preferred if upstream data is large, but requires a shuffle):

df
   .repartition(1)
   .write.format("com.databricks.spark.csv")
   .option("header", "true")
   .save("mydata.csv")

or coalesce:

df
   .coalesce(1)
   .write.format("com.databricks.spark.csv")
   .option("header", "true")
   .save("mydata.csv")

data frame before saving:

All data will be written to mydata.csv/part-00000. Before you use this option be sure you understand what is going on and what is the cost of transferring all data to a single worker. If you use distributed file system with replication, data will be transfered multiple times – first fetched to a single worker and subsequently distributed over storage nodes.

Alternatively you can leave your code as it is and use general purpose tools like cat or HDFS getmerge to simply merge all the parts afterwards.

More Related Contents:

  1. How do I skip a header from CSV files in Spark?
  2. Specifying the filename when saving a DataFrame as a CSV [duplicate]
  3. Provide schema while reading csv file as a dataframe
  4. Provide schema while reading csv file as a dataframe in Scala Spark
  5. How to bucket the range of values from a column and count how many values fall into each interval in scala?
  6. How to store custom objects in Dataset?
  7. How do I detect if a Spark DataFrame has a column
  8. How to avoid duplicate columns after join?
  9. Flattening Rows in Spark
  10. How to split a dataframe into dataframes with same column values?
  11. How to use regex to include/exclude some input files in sc.textFile?
  12. How to zip two (or more) DataFrame in Spark
  13. How to force DataFrame evaluation in Spark
  14. How to find spark RDD/Dataframe size?
  15. call of distinct and map together throws NPE in spark library
  16. How to get path to the uploaded file
  17. how to filter out a null value from spark dataframe
  18. Is there a reason not to use SparkContext.getOrCreate when writing a spark job?
  19. What are the various join types in Spark?
  20. Replace missing values with mean – Spark Dataframe
  21. How to calculate the size of dataframe in bytes in Spark?
  22. Why does partition parameter of SparkContext.textFile not take effect?
  23. Spark Scala list folders in directory
  24. Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)
  25. Spark-Monotonically increasing id not working as expected in dataframe?
  26. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  27. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  28. How to print the contents of RDD?
  29. Scala-Spark Dynamically call groupby and agg with parameter values
  30. Joining two dataframes without a common column

Leave a Comment