Spark column string replace when present in other column (row)

by

You could simply use regexp_replace

df5.withColumn("sentence_without_label", regexp_replace($"sentence" , lit($"label"), lit("" )))

or you can use simple udf function as below 

val df5 = spark.createDataFrame(Seq(
  ("Hi I heard about Spark", "Spark"),
  ("I wish Java could use case classes", "Java"),
  ("Logistic regression models are neat", "models")
)).toDF("sentence", "label")

val replace = udf((data: String , rep : String)=>data.replaceAll(rep, ""))

val res = df5.withColumn("sentence_without_label", replace($"sentence" , $"label"))

res.show()

Output: 

+-----------------------------------+------+------------------------------+
|sentence                           |label |sentence_without_label        |
+-----------------------------------+------+------------------------------+
|Hi I heard about Spark             |Spark |Hi I heard about              |
|I wish Java could use case classes |Java  |I wish  could use case classes|
|Logistic regression models are neat|models|Logistic regression  are neat |
+-----------------------------------+------+------------------------------+

More Related Contents:

  1. How to define and use a User-Defined Aggregate Function in Spark SQL?
  2. How can I pass extra parameters to UDFs in Spark SQL?
  3. SparkSQL: How to deal with null values in user defined function?
  4. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  5. About how to add a new column to an existing DataFrame with random values in Scala
  6. Spark UDAF with ArrayType as bufferSchema performance issues
  7. Derive multiple columns from a single column in a Spark DataFrame
  8. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  9. Spark – load CSV file as DataFrame?
  10. How to define partitioning of DataFrame?
  11. Case class equality in Apache Spark
  12. Dropping a nested column from Spark DataFrame
  13. How to read records in JSON format from Kafka using Structured Streaming?
  14. Spark dataframe write method writing many small files
  15. Filling gaps in timeseries Spark
  16. java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags
  17. How to use COGROUP for large datasets
  18. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  19. Parsing multiline records in Scala
  20. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  21. Create new Dataframe with empty/null field values
  22. Spark: Add column to dataframe conditionally
  23. Explanation of fold method of spark RDD
  24. How to explode an array into multiple columns in Spark
  25. Spark2.1.0 incompatible Jackson versions 2.7.6
  26. Coalesce reduces parallelism of entire stage (spark)
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment