spark broadcast variable Map giving null value

by

lit() return Column type, but map.get require the int type
you can do in this way

    val df: DataFrame = spark.sparkContext.parallelize(Range(0, 10000), 4).toDF("sentiment")
    val map = new util.HashMap[Int, Int]()
    map.put(1, 1)
    map.put(2, 2)
    map.put(3, 3)
    val bf: Broadcast[util.HashMap[Int, Int]] = spark.sparkContext.broadcast(map)
    df.rdd.map(x => {
      val num = x.getInt(0)
      (num, bf.value.get(num))
    }).toDF("key", "add_key").show(false)

More Related Contents:

  1. How to handle this scenario in spark? [closed]
  2. How to do aggregate functions, may columns and extract back
  3. How to flatten a struct in a Spark dataframe?
  4. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  5. Spark sql how to explode without losing null values
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. How can I update a broadcast variable in spark streaming?
  8. How to set timezone to UTC in Apache Spark?
  9. Change output filename prefix for DataFrame.write()
  10. How do I call a UDF on a Spark DataFrame using JAVA?
  11. Retain keys with null values while writing JSON in spark
  12. Reading data from Azure Blob with Spark
  13. How to use Column.isin in Java?
  14. Why does SparkSession execute twice for one action?
  15. Resolving dependency problems in Apache Spark
  16. Add JAR files to a Spark job – spark-submit
  17. Spark: How to map Python with Scala or Java User Defined Functions?
  18. Spark read file from S3 using sc.textFile (“s3n://…)
  19. java.lang.ClassCastException using lambda expressions in spark job on remote server
  20. Apache Spark – foreach Vs foreachPartition When to use What?
  21. Matrix Multiplication in Apache Spark [closed]
  22. How to save/insert each DStream into a permanent table
  23. TaskSchedulerImpl: Initial job has not accepted any resources;
  24. java.lang.NoClassDefFoundError: org/apache/spark/Logging
  25. Is gzip format supported in Spark?
  26. How can I force Spark to execute code?
  27. How to calculate the size of dataframe in bytes in Spark?
  28. How to save models from ML Pipeline to S3 or HDFS?
  29. Converting mysql table to spark dataset is very slow compared to same from csv file
  30. Running custom Java class in PySpark

Leave a Comment