Why does SparkSession execute twice for one action?

by

It happens because you don’t provide schema for DataFrameReader. As a result Spark has to eagerly scan data set to infer output schema.

Since mappedRdd is not cached it will be evaluated twice:

  • once for schema inference
  • once when you call data.show

If you want to prevent you should provide schema for reader (Scala syntax):

val schema: org.apache.spark.sql.types.StructType = ???
spark.read.schema(schema).json(mappedRdd)

More Related Contents:

  1. How to handle this scenario in spark? [closed]
  2. How to do aggregate functions, may columns and extract back
  3. How to flatten a struct in a Spark dataframe?
  4. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  5. Spark sql how to explode without losing null values
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. spark broadcast variable Map giving null value
  8. How to set timezone to UTC in Apache Spark?
  9. Change output filename prefix for DataFrame.write()
  10. How do I call a UDF on a Spark DataFrame using JAVA?
  11. Retain keys with null values while writing JSON in spark
  12. How to use Column.isin in Java?
  13. Add JAR files to a Spark job – spark-submit
  14. Spark: How to map Python with Scala or Java User Defined Functions?
  15. Read whole text files from a compression in Spark
  16. Spark read file from S3 using sc.textFile (“s3n://…)
  17. java.lang.ClassCastException using lambda expressions in spark job on remote server
  18. Spark Error – Unsupported class file major version
  19. How can I update a broadcast variable in spark streaming?
  20. Apache Spark – foreach Vs foreachPartition When to use What?
  21. Matrix Multiplication in Apache Spark [closed]
  22. What are the Spark transformations that causes a Shuffle?
  23. TaskSchedulerImpl: Initial job has not accepted any resources;
  24. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  25. Is gzip format supported in Spark?
  26. How to get element by Index in Spark RDD (Java)
  27. How to save models from ML Pipeline to S3 or HDFS?
  28. Converting mysql table to spark dataset is very slow compared to same from csv file
  29. Spark: get number of cluster cores programmatically
  30. Running custom Java class in PySpark

Leave a Comment