How to find spark RDD/Dataframe size?

by

If you are simply looking to count the number of rows in the rdd, do:

val distFile = sc.textFile(file)
println(distFile.count)

If you are interested in the bytes, you can use the SizeEstimator:

import org.apache.spark.util.SizeEstimator
println(SizeEstimator.estimate(distFile))

https://spark.apache.org/docs/latest/api/java/org/apache/spark/util/SizeEstimator.html

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. (Why) do we need to call cache or persist on a RDD
  4. Case class equality in Apache Spark
  5. Spark performance for Scala vs Python
  6. Stackoverflow due to long RDD Lineage
  7. Modify collection inside a Spark RDD foreach
  8. Parsing multiline records in Scala
  9. Explanation of fold method of spark RDD
  10. Why does partition parameter of SparkContext.textFile not take effect?
  11. Why does Spark RDD partition has 2GB limit for HDFS?
  12. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  13. How to sort an RDD in Scala Spark?
  14. How to transpose an RDD in Spark
  15. Automatically and Elegantly flatten DataFrame in Spark SQL
  16. How do I skip a header from CSV files in Spark?
  17. Spark extracting values from a Row
  18. How to pass -D parameter or environment variable to Spark job?
  19. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  20. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  21. MatchError while accessing vector column in Spark 2.0
  22. Spark / Scala: forward fill with last observation
  23. Function returns an empty List in Spark
  24. How to compare two dataframe and print columns that are different in scala
  25. Operate on neighbor elements in RDD in Spark
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to get keys and values from MapType column in SparkSQL DataFrame
  28. Addition of two RDD[mllib.linalg.Vector]’s
  29. Spark column string replace when present in other column (row)
  30. How to vectorize DataFrame columns for ML algorithms?

Leave a Comment