Spark dataframe write method writing many small files

by

you have to repartiton your DataFrame to match the partitioning of the DataFrameWriter

Try this:

df
.repartition($"date")
.write.mode(SaveMode.Append)
.partitionBy("date")
.parquet(s"$path")

More Related Contents:

  1. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  2. How MapReduce work in Apache Spark and Scala?
  3. Write to multiple outputs by key Spark – one Spark job
  4. How does HashPartitioner work?
  5. Spark 2.0 Dataset vs DataFrame
  6. Spark – load CSV file as DataFrame?
  7. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  8. How to define partitioning of DataFrame?
  9. Case class equality in Apache Spark
  10. Why accesing DataFrame from UDF results in NullPointerException?
  11. Dropping a nested column from Spark DataFrame
  12. How to read records in JSON format from Kafka using Structured Streaming?
  13. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  14. Scala spark, listbuffer is empty
  15. Change nullable property of column in spark dataframe
  16. Reading DataFrame from partitioned parquet file
  17. How to convert unix timestamp to date in Spark
  18. Perform a typed join in Scala with Spark Datasets
  19. Apache Spark: Get number of records per partition
  20. Left Anti join in Spark?
  21. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  22. DataFrame-ified zipWithIndex
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. How to sort by column in descending order in Spark SQL?
  26. How to explode an array into multiple columns in Spark
  27. Why is join not possible after show operator?
  28. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment