multiple conditions for filter in spark data frames

by

Instead of:

df2 = df1.filter("Status=2" || "Status =3")

Try:

df2 = df1.filter($"Status" === 2 || $"Status" === 3)

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Find maximum row per group in Spark DataFrame
  3. How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?
  4. While writing to hdfs path getting error java.io.IOException: Failed to rename
  5. Multiple Aggregate operations on the same column of a spark dataframe
  6. Spark SQL – load data with JDBC using SQL statement, not table name
  7. How to access element of a VectorUDT column in a Spark DataFrame?
  8. Spark Dataframe validating column names for parquet writes
  9. How to optimize partitioning when migrating data from JDBC source?
  10. Filtering a spark dataframe based on date
  11. How to save/insert each DStream into a permanent table
  12. Pyspark : forward fill with last observation for a DataFrame
  13. Spark DataFrame Schema Nullable Fields
  14. Spark load data and add filename as dataframe column
  15. PySpark: how to resample frequencies
  16. Why does Spark think this is a cross / Cartesian join
  17. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  18. Spark SQL broadcast hash join
  19. What is the difference between Apache Spark SQLContext vs HiveContext?
  20. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  21. pyspark: count distinct over a window
  22. Spark lists all leaf node even in partitioned data
  23. Array Intersection in Spark SQL
  24. PySpark: How to fillna values in dataframe for specific columns?
  25. How to get Kafka offsets for structured query for manual and reliable offset management?
  26. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  27. How to calculate Median in spark sqlContext for column of data type double
  28. Spark: disk I/O on stage boundaries explanation
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment