How to handle this scenario in spark? [closed]

by

Join both datasets based on company_id and select all columns from second dataset. Code should look something as below: (Not Tested)

Dataset<Row> finalDf = firstDataset.join(secondDataset 
,firstDataset.col("companyId").equalTo(secondDataset.col("companyid"), 
"inner").select(secondDataset .col("*))
finalDF.show()

More Related Contents:

  1. How to do aggregate functions, may columns and extract back
  2. How to flatten a struct in a Spark dataframe?
  3. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  4. Spark sql how to explode without losing null values
  5. Spark Strutured Streaming automatically converts timestamp to local time
  6. spark broadcast variable Map giving null value
  7. How to set timezone to UTC in Apache Spark?
  8. Change output filename prefix for DataFrame.write()
  9. How do I call a UDF on a Spark DataFrame using JAVA?
  10. Retain keys with null values while writing JSON in spark
  11. How to use Column.isin in Java?
  12. Why does SparkSession execute twice for one action?
  13. Resolving dependency problems in Apache Spark
  14. Add JAR files to a Spark job – spark-submit
  15. Spark: How to map Python with Scala or Java User Defined Functions?
  16. Read whole text files from a compression in Spark
  17. Spark read file from S3 using sc.textFile (“s3n://…)
  18. java.lang.ClassCastException using lambda expressions in spark job on remote server
  19. Spark Error – Unsupported class file major version
  20. How can I update a broadcast variable in spark streaming?
  21. Apache Spark – foreach Vs foreachPartition When to use What?
  22. Matrix Multiplication in Apache Spark [closed]
  23. What are the Spark transformations that causes a Shuffle?
  24. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  25. Is gzip format supported in Spark?
  26. How to get element by Index in Spark RDD (Java)
  27. How to save models from ML Pipeline to S3 or HDFS?
  28. Converting mysql table to spark dataset is very slow compared to same from csv file
  29. Spark: get number of cluster cores programmatically
  30. Running custom Java class in PySpark

Leave a Comment