Skewed dataset join in Spark?

by

Pretty good article on how it can be done: https://datarus.wordpress.com/2015/05/04/fighting-the-skew-in-spark/

Short version:

  • Add random element to large RDD and create new join key with it
  • Add random element to small RDD using explode/flatMap to increase number of entries and create new join key
  • Join RDDs on new join key which will now be distributed better due to random seeding

More Related Contents:

  1. Including null values in an Apache Spark Join
  2. How to join query in mongodb?
  3. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  4. Perform a typed join in Scala with Spark Datasets
  5. Why is join not possible after show operator?
  6. How to query JSON data column using Spark DataFrames?
  7. How to do an update + join in PostgreSQL?
  8. Applying UDFs on GroupedData in PySpark (with functioning python example)
  9. How do I detect if a Spark DataFrame has a column
  10. How to change a dataframe column from String type to Double type in PySpark?
  11. How can PySpark be called in debug mode?
  12. Spark DataFrame: count distinct values of every column
  13. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  14. How to force DataFrame evaluation in Spark
  15. How to use a Scala class inside Pyspark
  16. How to add third-party Java JAR files for use in PySpark
  17. Filter Pyspark dataframe column with None value
  18. LEFT JOIN only first row
  19. call of distinct and map together throws NPE in spark library
  20. How to get path to the uploaded file
  21. How to find mean of grouped Vector columns in Spark SQL?
  22. Fill in null with previously known good value with pyspark
  23. Change output filename prefix for DataFrame.write()
  24. Which cluster type should I choose for Spark? [closed]
  25. rolling joins data.table in R
  26. Identify records in data frame A not contained in data frame B [closed]
  27. reduce result datasets into single dataset
  28. AWS Glue executor memory limit
  29. spark ssc.textFileStream is not streamining any files from directory
  30. What is an optimized way of joining large tables in Spark SQL

Leave a Comment