How to calculate the size of dataframe in bytes in Spark?

by

Usingspark.sessionState.executePlan(df.queryExecution.logical).optimizedPlan.stats(spark.sessionState.conf).sizeInBytes we can get the size of actual Dataframe once its loaded into memory. Check the below code.

scala> val df = spark.read.format("orc").load("/tmp/srinivas/")
df: org.apache.spark.sql.DataFrame = [channelGrouping: string, clientId: string ... 75 more fields]

scala> import org.apache.commons.io.FileUtils
import org.apache.commons.io.FileUtils

scala> val bytes = spark.sessionState.executePlan(df.queryExecution.logical).optimizedPlan.stats(spark.sessionState.conf).sizeInBytes
bytes: BigInt = 763275709

scala> FileUtils.byteCountToDisplaySize(bytes.toLong)
res5: String = 727 MB

scala> import sys.process._
import sys.process._

scala> "hdfs dfs -ls -h /tmp/srinivas/".!
Found 2 items
-rw-r-----   3 svcmxns hdfs          0 2020-04-20 01:46 /tmp/srinivas/_SUCCESS
-rw-r-----   3 svcmxns hdfs    727.4 M 2020-04-20 01:46 /tmp/srinivas/part-00000-9d0b72ea-f617-4092-ae27-d36400c17917-c000.snappy.orc
res6: Int = 0

val bytes = spark.sessionState.executePlan(df.queryExecution.logical).optimizedPlan.stats(spark.sessionState.conf).sizeInBytes
    val dataSize = bytes.toLong
    val numPartitions = (bytes.toLong./(1024.0)./(1024.0)./(10240)).ceil.toInt // May be you can change or modify this to get required partitions.

    df.repartition(if(numPartitions == 0) 1 else numPartitions)
      .[...]

Edit - 1 : Please use the following logic as per your Spark versions.

Spark 2.4

val bytes = spark.sessionState.executePlan(df.queryExecution.logical).optimizedPlan.stats(spark.sessionState.conf).sizeInBytes

Spark 2.3

val bytes = spark.sessionState.executePlan(df.queryExecution.logical).optimizedPlan.stats.sizeInBytes

PySpark

spark._jsparkSession.sessionState().executePlan(df._jdf.queryExecution().logical()).optimizedPlan().stats().sizeInBytes()

More Related Contents:

  1. Spark DataFrame: does groupBy after orderBy maintain that order?
  2. How to convert rdd object to dataframe in spark
  3. Spark 2.0 Dataset vs DataFrame
  4. How to define and use a User-Defined Aggregate Function in Spark SQL?
  5. Automatically and Elegantly flatten DataFrame in Spark SQL
  6. Spark extracting values from a Row
  7. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  8. SparkSQL: How to deal with null values in user defined function?
  9. How to aggregate values into collection after groupBy?
  10. How to read records in JSON format from Kafka using Structured Streaming?
  11. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  12. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  13. Processing multiple files as independent RDD’s in parallel
  14. MatchError while accessing vector column in Spark 2.0
  15. How to save DataFrame directly to Hive?
  16. Spark / Scala: forward fill with last observation
  17. About how to add a new column to an existing DataFrame with random values in Scala
  18. How to use Column.isin with list?
  19. How to compare two dataframe and print columns that are different in scala
  20. Perform a typed join in Scala with Spark Datasets
  21. Apache Spark: Get number of records per partition
  22. Renaming column names of a DataFrame in Spark Scala
  23. DataFrame-ified zipWithIndex
  24. Derive multiple columns from a single column in a Spark DataFrame
  25. How to save a spark DataFrame as csv on disk?
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to sort by column in descending order in Spark SQL?
  28. Exploding nested Struct in Spark dataframe
  29. How to get keys and values from MapType column in SparkSQL DataFrame
  30. How to create DataFrame from Scala’s List of Iterables?

Leave a Comment