How to drop all columns with null values in a PySpark DataFrame?

by

Here is one possible approach for dropping all columns that have NULL values: See here for the source on the code of counting NULL values per column.

import pyspark.sql.functions as F

# Sample data
df = pd.DataFrame({'x1': ['a', '1', '2'],
                   'x2': ['b', None, '2'],
                   'x3': ['c', '0', '3'] })
df = sqlContext.createDataFrame(df)
df.show()

def drop_null_columns(df):
    """
    This function drops all columns which contain null values.
    :param df: A PySpark DataFrame
    """
    null_counts = df.select([F.count(F.when(F.col(c).isNull(), c)).alias(c) for c in df.columns]).collect()[0].asDict()
    to_drop = [k for k, v in null_counts.items() if v > 0]
    df = df.drop(*to_drop)
    return df

# Drops column b2, because it contains null values
drop_null_columns(df).show()

Before:

+---+----+---+
| x1|  x2| x3|
+---+----+---+
|  a|   b|  c|
|  1|null|  0|
|  2|   2|  3|
+---+----+---+

After:

+---+---+
| x1| x3|
+---+---+
|  a|  c|
|  1|  0|
|  2|  3|
+---+---+

Hope this helps!

More Related Contents:

  1. Spark Equivalent of IF Then ELSE
  2. How to add a constant column in a Spark DataFrame?
  3. How to split Vector into columns – using PySpark
  4. Spark Dataframe distinguish columns with duplicated name
  5. How to change dataframe column names in pyspark?
  6. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  7. Explode in PySpark
  8. Retrieve top n in each group of a DataFrame in pyspark
  9. Pyspark: explode json in column to multiple columns
  10. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  11. Dividing complex rows of dataframe to simple rows in Pyspark
  12. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  13. Why is Apache-Spark – Python so slow locally as compared to pandas?
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. Spark add new column to dataframe with value from previous row
  17. PySpark create new column with mapping from a dict
  18. Pyspark: Replacing value in a column by searching a dictionary
  19. Create Spark DataFrame. Can not infer schema for type
  20. Pivot String column on Pyspark Dataframe
  21. Using UDF ignores condition in when
  22. Create single row dataframe from list of list PySpark
  23. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. When to cache a DataFrame?
  27. How to add suffix and prefix to all columns in python/pyspark dataframe
  28. Pyspark changing type of column from date to string
  29. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment