How to exclude multiple columns in Spark dataframe in Python

by

In PySpark 2.1.0 method drop supports multiple columns:

PySpark 2.0.2:

DataFrame.drop(col)

PySpark 2.1.0:

DataFrame.drop(*cols)

Example:

df.drop('col1', 'col2')

or using the * operator as

df.drop(*['col1', 'col2'])

More Related Contents:

  1. How to access element of a VectorUDT column in a Spark DataFrame?
  2. How to loop through each row of dataFrame in pyspark
  3. Why does Spark think this is a cross / Cartesian join
  4. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  5. Finding duplicates from large data set using Apache Spark
  6. How to make good reproducible Apache Spark examples
  7. Find maximum row per group in Spark DataFrame
  8. How to add a constant column in a Spark DataFrame?
  9. Multiple Aggregate operations on the same column of a spark dataframe
  10. Spark Dataframe distinguish columns with duplicated name
  11. How do I add a new column to a Spark DataFrame (using PySpark)?
  12. Spark Dataframe validating column names for parquet writes
  13. Updating a dataframe column in spark
  14. Dividing complex rows of dataframe to simple rows in Pyspark
  15. How to save/insert each DStream into a permanent table
  16. Pyspark : forward fill with last observation for a DataFrame
  17. Spark load data and add filename as dataframe column
  18. Spark add new column to dataframe with value from previous row
  19. PySpark: how to resample frequencies
  20. Pyspark: Replacing value in a column by searching a dictionary
  21. Create Spark DataFrame. Can not infer schema for type
  22. Pivot String column on Pyspark Dataframe
  23. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  24. Pyspark filter dataframe by columns of another dataframe
  25. Rename more than one column using withColumnRenamed
  26. spark dataframe drop duplicates and keep first
  27. pyspark: count distinct over a window
  28. PySpark: How to fillna values in dataframe for specific columns?
  29. Filtering DataFrame using the length of a column
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment