How to control partition size in Spark SQL

by

Spark < 2.0:

You can use Hadoop configuration options:

  • mapred.min.split.size.
  • mapred.max.split.size

as well as HDFS block size to control partition size for filesystem based formats*.

val minSplit: Int = ???
val maxSplit: Int = ???

sc.hadoopConfiguration.setInt("mapred.min.split.size", minSplit)
sc.hadoopConfiguration.setInt("mapred.max.split.size", maxSplit)

Spark 2.0+:

You can use spark.sql.files.maxPartitionBytes configuration:

spark.conf.set("spark.sql.files.maxPartitionBytes", maxSplit)

In both cases these values may not be in use by a specific data source API so you should always check documentation / implementation details of the format you use.

* Other input formats can use different settings. See for example

Furthermore Datasets created from RDDs will inherit partition layout from their parents.

Similarly bucketed tables will use bucket layout defined in the metastore with 1:1 relationship between bucket and Dataset partition.

More Related Contents:

  1. How to optimize partitioning when migrating data from JDBC source?
  2. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  3. Partitioning in spark while reading from RDBMS via JDBC
  4. Avoid performance impact of a single partition mode in Spark window functions
  5. How to connect to remote hive server from spark [duplicate]
  6. Use collect_list and collect_set in Spark SQL
  7. What is the difference between Apache Spark SQLContext vs HiveContext?
  8. Spark lists all leaf node even in partitioned data
  9. How to calculate Median in spark sqlContext for column of data type double
  10. How to melt Spark DataFrame?
  11. Spark SQL: apply aggregate functions to a list of columns
  12. Unpivot in spark-sql/pyspark
  13. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  14. DataFrame join optimization – Broadcast Hash Join
  15. Overwrite specific partitions in spark dataframe write method
  16. Spark Dataframe validating column names for parquet writes
  17. How to check if spark dataframe is empty?
  18. Pyspark : forward fill with last observation for a DataFrame
  19. Multiple Spark applications with HiveContext
  20. multiple conditions for filter in spark data frames
  21. How do I add an persistent column of row ids to Spark DataFrame?
  22. Rename more than one column using withColumnRenamed
  23. Convert null values to empty array in Spark DataFrame
  24. DataFrame partitionBy to a single Parquet file (per partition)
  25. Temp table caching with spark-sql
  26. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  27. Write Spark dataframe as CSV with partitions
  28. What does “Correlated scalar subqueries must be Aggregated” mean?
  29. How to improve broadcast Join speed with between condition in Spark
  30. Spark SQL referencing attributes of UDT

Leave a Comment