Filter Pyspark dataframe column with None value

by

You can use Column.isNull / Column.isNotNull:

df.where(col("dt_mvmt").isNull())

df.where(col("dt_mvmt").isNotNull())

If you want to simply drop NULL values you can use na.drop with subset argument:

df.na.drop(subset=["dt_mvmt"])

Equality based comparisons with NULL won’t work because in SQL NULL is undefined so any attempt to compare it with another value returns NULL:

sqlContext.sql("SELECT NULL = NULL").show()
## +-------------+
## |(NULL = NULL)|
## +-------------+
## |         null|
## +-------------+


sqlContext.sql("SELECT NULL != NULL").show()
## +-------------------+
## |(NOT (NULL = NULL))|
## +-------------------+
## |               null|
## +-------------------+

The only valid method to compare value with NULL is IS / IS NOT which are equivalent to the isNull / isNotNull method calls.

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. Spark Dataframe distinguish columns with duplicated name
  3. Pyspark: Split multiple array columns into rows
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to change a dataframe column from String type to Double type in PySpark?
  6. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  7. Retrieve top n in each group of a DataFrame in pyspark
  8. Updating a dataframe column in spark
  9. Dividing complex rows of dataframe to simple rows in Pyspark
  10. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  11. How to explode multiple columns of a dataframe in pyspark
  12. Spark add new column to dataframe with value from previous row
  13. PySpark: multiple conditions in when clause
  14. Pyspark: Replacing value in a column by searching a dictionary
  15. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  16. Create Spark DataFrame. Can not infer schema for type
  17. Pivot String column on Pyspark Dataframe
  18. Filtering DataFrame using the length of a column
  19. Multiple condition filter on dataframe
  20. How to return a “Tuple type” in a UDF in PySpark?
  21. Load CSV file with Spark
  22. Does spark predicate pushdown work with JDBC?
  23. Rename nested field in spark dataframe
  24. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  25. Passing a data frame column and external list to udf under withColumn
  26. Best way to get the max value in a Spark dataframe column
  27. Apache Spark Python Cosine Similarity over DataFrames
  28. GroupBy column and filter rows with maximum value in Pyspark
  29. When to cache a DataFrame?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment