Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?

by

This happens because Spark tries to do Broadcast Hash Join and one of the DataFrames is very large, so sending it consumes much time.

You can:

  1. Set higher spark.sql.broadcastTimeout to increase timeout – spark.conf.set("spark.sql.broadcastTimeout", newValueForExample36000)
  2. persist() both DataFrames, then Spark will use Shuffle Join – reference from here

PySpark

In PySpark, you can set the config when you build the spark context in the following manner:

spark = SparkSession
  .builder
  .appName("Your App")
  .config("spark.sql.broadcastTimeout", "36000")
  .getOrCreate()

More Related Contents:

  1. Perform a typed join in Scala with Spark Datasets
  2. Why is join not possible after show operator?
  3. Spark 2.0 Dataset vs DataFrame
  4. Spark – load CSV file as DataFrame?
  5. How to define partitioning of DataFrame?
  6. Automatically and Elegantly flatten DataFrame in Spark SQL
  7. Spark extracting values from a Row
  8. SparkSQL: How to deal with null values in user defined function?
  9. How to aggregate values into collection after groupBy?
  10. Dropping a nested column from Spark DataFrame
  11. How to read records in JSON format from Kafka using Structured Streaming?
  12. Including null values in an Apache Spark Join
  13. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  14. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  15. MatchError while accessing vector column in Spark 2.0
  16. Spark / Scala: forward fill with last observation
  17. Filling gaps in timeseries Spark
  18. How to use COGROUP for large datasets
  19. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  20. How to compare two dataframe and print columns that are different in scala
  21. Apache Spark: Get number of records per partition
  22. Renaming column names of a DataFrame in Spark Scala
  23. DataFrame-ified zipWithIndex
  24. Derive multiple columns from a single column in a Spark DataFrame
  25. How to save a spark DataFrame as csv on disk?
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to sort by column in descending order in Spark SQL?
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. How to create DataFrame from Scala’s List of Iterables?
  30. How to create a Dataset of Maps?

Leave a Comment