Pyspark replace strings in Spark dataframe column

by

For Spark 1.5 or later, you can use the functions package:

from pyspark.sql.functions import *
newDf = df.withColumn('address', regexp_replace('address', 'lane', 'ln'))

Quick explanation:

  • The function withColumn is called to add (or replace, if the name exists) a column to the data frame.
  • The function regexp_replace will generate a new column by replacing all substrings that match the pattern.

More Related Contents:

  1. How to split Vector into columns – using PySpark
  2. How to use JDBC source to write and read data in (Py)Spark?
  3. Applying UDFs on GroupedData in PySpark (with functioning python example)
  4. Pyspark: Split multiple array columns into rows
  5. creating spark data structure from multiline record
  6. How to change a dataframe column from String type to Double type in PySpark?
  7. Retrieve top n in each group of a DataFrame in pyspark
  8. Pyspark: explode json in column to multiple columns
  9. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  10. How to use a Scala class inside Pyspark
  11. How to add third-party Java JAR files for use in PySpark
  12. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  13. Filter Pyspark dataframe column with None value
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. Convert spark DataFrame column to python list
  17. PySpark: multiple conditions in when clause
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Load CSV file with PySpark
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. Cannot find col function in pyspark
  22. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  23. ImportError: No module named numpy on spark workers
  24. How to pass a constant value to Python UDF?
  25. How to zip two array columns in Spark SQL
  26. When to cache a DataFrame?
  27. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  28. PySpark Throwing error Method __getnewargs__([]) does not exist
  29. PySpark Evaluation
  30. What is the difference between spark-submit and pyspark?

Leave a Comment