What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]

by

Question : I was wondering what may cause this exception to be thrown?

Answer :

spark.sql.broadcastTimeout 300 Timeout in seconds for the broadcast
wait time in broadcast joins

spark.network.timeout 120s Default timeout for all network interactions.. spark.network.timeout (spark.rpc.askTimeout), spark.sql.broadcastTimeout,
spark.kryoserializer.buffer.max(if you are using kryo
serialization), etc. are tuned with larger-than-default values in
order to handle complex queries. You can start with these values and
adjust accordingly to your SQL workloads.

Note : Doc says that

The following options(see spark.sql. properties) can also be used to tune the performance of query execution. It is possible that these options will be deprecated in future release as more optimizations are performed automatically.*

Also,for your better understanding you can see BroadCastHashJoin where execute method is trigger point for the above stack trace.

protected override def doExecute(): RDD[Row] = {
    val broadcastRelation = Await.result(broadcastFuture, timeout)

    streamedPlan.execute().mapPartitions { streamedIter =>
      hashJoin(streamedIter, broadcastRelation.value)
    }
  }

More Related Contents:

  1. Spark 2.0 Dataset vs DataFrame
  2. Spark – load CSV file as DataFrame?
  3. How to define partitioning of DataFrame?
  4. Automatically and Elegantly flatten DataFrame in Spark SQL
  5. Spark extracting values from a Row
  6. SparkSQL: How to deal with null values in user defined function?
  7. How to aggregate values into collection after groupBy?
  8. Dropping a nested column from Spark DataFrame
  9. How to read records in JSON format from Kafka using Structured Streaming?
  10. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. How to save DataFrame directly to Hive?
  15. Spark / Scala: forward fill with last observation
  16. Filling gaps in timeseries Spark
  17. How to use COGROUP for large datasets
  18. How to compare two dataframe and print columns that are different in scala
  19. Perform a typed join in Scala with Spark Datasets
  20. Apache Spark: Get number of records per partition
  21. Renaming column names of a DataFrame in Spark Scala
  22. DataFrame-ified zipWithIndex
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. What is going wrong with `unionAll` of Spark `DataFrame`?
  26. How to sort by column in descending order in Spark SQL?
  27. Exploding nested Struct in Spark dataframe
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. Why is join not possible after show operator?
  30. Difference between two rows in Spark dataframe

Leave a Comment