Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion

by

By referencing the object containing your broadcast variable in your map lambda, Spark will attempt to serialize the whole object and ship it to workers. Since the object contains a reference to the SparkContext, you get the error. Instead of this:

pairs = distinct_users_projected.map(lambda x: (x.user_id, pt.broadcast_products_lookup_map.value[x.Prod_ID]))

Try this:

bcast = pt.broadcast_products_lookup_map
pairs = distinct_users_projected.map(lambda x: (x.user_id, bcast.value[x.Prod_ID]))

The latter avoids the reference to the object (pt) so that Spark only needs to ship the broadcast variable.

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Efficient string matching in Apache Spark
  3. Load CSV file with Spark
  4. How to link PyCharm with PySpark?
  5. How to turn off INFO logging in Spark?
  6. How do I add a new column to a Spark DataFrame (using PySpark)?
  7. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  8. importing pyspark in python shell
  9. collect_list by preserving order based on another variable
  10. Rename nested field in spark dataframe
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. How to determine if object is a valid key-value pair in PySpark
  13. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  14. A list as a key for PySpark’s reduceByKey
  15. Best way to get the max value in a Spark dataframe column
  16. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  17. Pyspark replace strings in Spark dataframe column
  18. Apache Spark Python Cosine Similarity over DataFrames
  19. Pyspark: Parse a column of json strings
  20. GroupBy column and filter rows with maximum value in Pyspark
  21. How do I set the driver’s python version in spark?
  22. Add Jar to standalone pyspark
  23. PySpark: match the values of a DataFrame column against another DataFrame column
  24. Spark mllib predicting weird number or NaN
  25. Multiple condition filter on dataframe
  26. How to return a “Tuple type” in a UDF in PySpark?
  27. How to connect HBase and Spark using Python?
  28. Pyspark 2.4.0, read avro from kafka with read stream – Python
  29. How to pivot on multiple columns in Spark SQL?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment