Specifying the filename when saving a DataFrame as a CSV [duplicate]

by

It’s not possible to do it directly in Spark’s save

Spark uses Hadoop File Format, which requires data to be partitioned – that’s why you have part- files. You can easily change filename after processing just like in this question

In Scala it will look like:

import org.apache.hadoop.fs._
val fs = FileSystem.get(sc.hadoopConfiguration)
val file = fs.globStatus(new Path("path/file.csv/part*"))(0).getPath().getName()

fs.rename(new Path("csvDirectory/" + file), new Path("mydata.csv"))
fs.delete(new Path("mydata.csv-temp"), true)

or just:

import org.apache.hadoop.fs._
val fs = FileSystem.get(sc.hadoopConfiguration)
fs.rename(new Path("csvDirectory/data.csv/part-0000"), new Path("csvDirectory/newData.csv"))

Edit: As mentioned in comments, you can also write your own OutputFormat, please see documents for information about this approach to set file name

More Related Contents:

  1. Write single CSV file using spark-csv
  2. How do I skip a header from CSV files in Spark?
  3. Spark performance for Scala vs Python
  4. How to set hadoop configuration values from pyspark
  5. How to define and use a User-Defined Aggregate Function in Spark SQL?
  6. (Why) do we need to call cache or persist on a RDD
  7. Automatically and Elegantly flatten DataFrame in Spark SQL
  8. Spark extracting values from a Row
  9. How do I convert csv file to rdd
  10. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  11. Provide schema while reading csv file as a dataframe
  12. How to flatmap a nested Dataframe in Spark
  13. How to save DataFrame directly to Hive?
  14. About how to add a new column to an existing DataFrame with random values in Scala
  15. How to use Column.isin with list?
  16. Function returns an empty List in Spark
  17. How to compare two dataframe and print columns that are different in scala
  18. Reading csv files with quoted fields containing embedded commas
  19. Operate on neighbor elements in RDD in Spark
  20. Customize SparkContext using sparkConf.set(..) when using spark-shell
  21. Renaming column names of a DataFrame in Spark Scala
  22. What is going wrong with `unionAll` of Spark `DataFrame`?
  23. Spark: long delay between jobs
  24. How to get keys and values from MapType column in SparkSQL DataFrame
  25. How to compute cumulative sum using Spark
  26. Addition of two RDD[mllib.linalg.Vector]’s
  27. Add Number of days column to Date Column in same dataframe for Spark Scala App
  28. How to print the contents of RDD?
  29. Scala-Spark Dynamically call groupby and agg with parameter values
  30. Joining two dataframes without a common column

Leave a Comment