Filtering a spark dataframe based on date

by

The following solutions are applicable since spark 1.5 :

For lower than :

// filter data where the date is lesser than 2015-03-14
data.filter(data("date").lt(lit("2015-03-14")))

For greater than :

// filter data where the date is greater than 2015-03-14
data.filter(data("date").gt(lit("2015-03-14")))

For equality, you can use either equalTo or === :

data.filter(data("date") === lit("2015-03-14"))

If your DataFrame date column is of type StringType, you can convert it using the to_date function :

// filter data where the date is greater than 2015-03-14
data.filter(to_date(data("date")).gt(lit("2015-03-14")))

You can also filter according to a year using the year function :

// filter data where year is greater or equal to 2016
data.filter(year($"date").geq(lit(2016)))

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Using a column value as a parameter to a spark DataFrame function
  3. Split Spark Dataframe string column into multiple columns
  4. While writing to hdfs path getting error java.io.IOException: Failed to rename
  5. Multiple Aggregate operations on the same column of a spark dataframe
  6. Partitioning in spark while reading from RDBMS via JDBC
  7. How to control partition size in Spark SQL
  8. How to import multiple csv files in a single load?
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. Spark DataFrame: count distinct values of every column
  11. How to split a list to multiple columns in Pyspark?
  12. Use collect_list and collect_set in Spark SQL
  13. What is the maximum size for a broadcast object in Spark?
  14. How to find mean of grouped Vector columns in Spark SQL?
  15. Generate a Spark StructType / Schema from a case class
  16. How to loop through each row of dataFrame in pyspark
  17. How does createOrReplaceTempView work in Spark?
  18. Fill in null with previously known good value with pyspark
  19. How to group by common element in array?
  20. PySpark – get row number for each row in a group
  21. Keep only duplicates from a DataFrame regarding some field
  22. Using windowing functions in Spark
  23. How can I access python variable in Spark SQL?
  24. reduce result datasets into single dataset
  25. How to exclude multiple columns in Spark dataframe in Python
  26. How to get Kafka offsets for structured query for manual and reliable offset management?
  27. How to use groupBy to collect rows into a map?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment