Filter spark DataFrame on string contains

by

You can use contains (this works with an arbitrary sequence):

df.filter($"foo".contains("bar"))

like (SQL like with SQL simple regular expression whith _ matching an arbitrary character and % matching an arbitrary sequence):

df.filter($"foo".like("bar"))

or rlike (like with Java regular expressions):

df.filter($"foo".rlike("bar"))

depending on your requirements. LIKE and RLIKE should work with SQL expressions as well.

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. How to zip two (or more) DataFrame in Spark
  11. Joining Spark dataframes on the key
  12. Apache Spark how to append new column from list/array to Spark dataframe
  13. Create new Dataframe with empty/null field values
  14. Provide schema while reading csv file as a dataframe in Scala Spark
  15. Renaming column names of a DataFrame in Spark Scala
  16. Derive multiple columns from a single column in a Spark DataFrame
  17. Replace missing values with mean – Spark Dataframe
  18. What is going wrong with `unionAll` of Spark `DataFrame`?
  19. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  20. How to get keys and values from MapType column in SparkSQL DataFrame
  21. Spark DataFrames when udf functions do not accept large enough input variables
  22. Flattening Rows in Spark
  23. How to flatmap a nested Dataframe in Spark
  24. How to write unit tests in Spark 2.0+?
  25. how to filter out a null value from spark dataframe
  26. What are the various join types in Spark?
  27. How to calculate the size of dataframe in bytes in Spark?
  28. Exploding nested Struct in Spark dataframe
  29. Add Number of days column to Date Column in same dataframe for Spark Scala App
  30. Spark Dataframe Random UUID changes after every transformation/action

Leave a Comment