Joining two dataframes without a common column

by

add an index column to both dataframe using the below code

df1.withColumn("id1",monotonicallyIncreasingId)
df2.withColumn("id2",monotonicallyIncreasingId)

then join both the dataframes using the below code and drop the index column

df1.join(df2,col("id1")===col("id2"),"inner")
   .drop("id1","id2")

More Related Contents:

  1. (Why) do we need to call cache or persist on a RDD
  2. How to load local file in sc.textFile, instead of HDFS
  3. Automatically and Elegantly flatten DataFrame in Spark SQL
  4. How do I skip a header from CSV files in Spark?
  5. Spark extracting values from a Row
  6. Stratified sampling in Spark
  7. How do I convert csv file to rdd
  8. How to pass -D parameter or environment variable to Spark job?
  9. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  10. Append a column to Data Frame in Apache Spark 1.3
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  15. Spark / Scala: forward fill with last observation
  16. How to convert unix timestamp to date in Spark
  17. Function returns an empty List in Spark
  18. Apache Spark: Get number of records per partition
  19. Operate on neighbor elements in RDD in Spark
  20. Renaming column names of a DataFrame in Spark Scala
  21. Derive multiple columns from a single column in a Spark DataFrame
  22. How to save a spark DataFrame as csv on disk?
  23. How to process multi line input records in Spark
  24. How to sort by column in descending order in Spark SQL?
  25. Spark: long delay between jobs
  26. Addition of two RDD[mllib.linalg.Vector]’s
  27. How to print the contents of RDD?
  28. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  29. How to create DataFrame from Scala’s List of Iterables?
  30. Scala-Spark Dynamically call groupby and agg with parameter values

Leave a Comment