Pyspark filter dataframe by columns of another dataframe

by

Left anti join is what you’re looking for:

df1.join(df2, ["userid", "group"], "leftanti")

but the same thing can be done with left outer join:

(df1
    .join(df2, ["userid", "group"], "leftouter")
    .where(df2["pick"].isNull())
    .drop(df2["pick"]))

More Related Contents:

  1. How to run independent transformations in parallel using PySpark?
  2. How to make good reproducible Apache Spark examples
  3. How to pivot Spark DataFrame?
  4. How to add a constant column in a Spark DataFrame?
  5. Spark Dataframe distinguish columns with duplicated name
  6. How do I add a new column to a Spark DataFrame (using PySpark)?
  7. How to change a dataframe column from String type to Double type in PySpark?
  8. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. Retrieve top n in each group of a DataFrame in pyspark
  11. Filter Spark DataFrame based on another DataFrame that specifies denylist criteria
  12. Updating a dataframe column in spark
  13. Dividing complex rows of dataframe to simple rows in Pyspark
  14. Filter Pyspark dataframe column with None value
  15. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  16. How to explode multiple columns of a dataframe in pyspark
  17. Spark add new column to dataframe with value from previous row
  18. PySpark: multiple conditions in when clause
  19. How to loop through each row of dataFrame in pyspark
  20. Why does Spark think this is a cross / Cartesian join
  21. Pyspark: Replacing value in a column by searching a dictionary
  22. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  23. Create Spark DataFrame. Can not infer schema for type
  24. Pivot String column on Pyspark Dataframe
  25. spark dataframe drop duplicates and keep first
  26. How to exclude multiple columns in Spark dataframe in Python
  27. Multiple condition filter on dataframe
  28. How to return a “Tuple type” in a UDF in PySpark?
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. SparkSQL on pyspark: how to generate time series?

Leave a Comment