Spark: subtract two DataFrames

by

According to the Scala API docs, doing:

dataFrame1.except(dataFrame2)

will return a new DataFrame containing rows in dataFrame1 but not in dataframe2.

More Related Contents:

  1. Unable to fetch the value of Println in apache spark
  2. Spark – repartition() vs coalesce()
  3. Spark SQL: apply aggregate functions to a list of columns
  4. How do I split an RDD into two or more RDDs?
  5. Difference between DataFrame, Dataset, and RDD in Spark
  6. DataFrame join optimization – Broadcast Hash Join
  7. What does “Stage Skipped” mean in Apache Spark web UI?
  8. How to access element of a VectorUDT column in a Spark DataFrame?
  9. Which operations preserve RDD order?
  10. Difference between DataFrame, Dataset, and RDD in Spark
  11. Is groupByKey ever preferred over reduceByKey
  12. Default Partitioning Scheme in Spark
  13. How DAG works under the covers in RDD?
  14. Spark parquet partitioning : Large number of files
  15. What is the maximum size for a broadcast object in Spark?
  16. How to loop through each row of dataFrame in pyspark
  17. Why does Spark think this is a cross / Cartesian join
  18. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  19. Why does sortBy transformation trigger a Spark job?
  20. How do I add an persistent column of row ids to Spark DataFrame?
  21. Apache spark dealing with case statements
  22. What is the difference between cache and persist?
  23. Convert null values to empty array in Spark DataFrame
  24. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  25. How to exclude multiple columns in Spark dataframe in Python
  26. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  27. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  28. Spark ALS predictAll returns empty
  29. Why is the fold action necessary in Spark?
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment