Change output filename prefix for DataFrame.write()

by

You cannot change the “part” prefix while using any of the standard output formats (like Parquet). See this snippet from ParquetRelation source code:

private val recordWriter: RecordWriter[Void, InternalRow] = {
  val outputFormat = {
    new ParquetOutputFormat[InternalRow]() {
      // ...
      override def getDefaultWorkFile(context: TaskAttemptContext, extension: String): Path = {
        // ..
        //  prefix is hard-coded here:
        new Path(path, f"part-r-$split%05d-$uniqueWriteJobId$bucketString$extension")
    }
  }
}

If you really must control the part file names, you’ll probably have to implement a custom FileOutputFormat and use one of Spark’s save methods that accept a FileOutputFormat class (e.g. saveAsHadoopFile).

More Related Contents:

  1. Spark Strutured Streaming automatically converts timestamp to local time
  2. Is gzip format supported in Spark?
  3. How to handle this scenario in spark? [closed]
  4. How to do aggregate functions, may columns and extract back
  5. Resolving dependency problems in Apache Spark
  6. Add JAR files to a Spark job – spark-submit
  7. How to flatten a struct in a Spark dataframe?
  8. Spark: How to map Python with Scala or Java User Defined Functions?
  9. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  10. Spark read file from S3 using sc.textFile (“s3n://…)
  11. Spark sql how to explode without losing null values
  12. How can I update a broadcast variable in spark streaming?
  13. Apache Spark – foreach Vs foreachPartition When to use What?
  14. Matrix Multiplication in Apache Spark [closed]
  15. spark broadcast variable Map giving null value
  16. What are the Spark transformations that causes a Shuffle?
  17. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  18. How to set timezone to UTC in Apache Spark?
  19. How can I force Spark to execute code?
  20. How do I call a UDF on a Spark DataFrame using JAVA?
  21. Retain keys with null values while writing JSON in spark
  22. How to save models from ML Pipeline to S3 or HDFS?
  23. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  24. How to use Column.isin in Java?
  25. Why does SparkSession execute twice for one action?
  26. How MapReduce work in Apache Spark and Scala?
  27. Scala equivalent of Java java.lang.Class Object
  28. How to aggregate values into collection after groupBy?
  29. How to read records in JSON format from Kafka using Structured Streaming?
  30. How can I use a Scala singleton object in Java?

Leave a Comment