Updating a dataframe column in spark

by

While you cannot modify a column as such, you may operate on a column and return a new DataFrame reflecting that change. For that you’d first create a UserDefinedFunction implementing the operation to apply and then selectively apply that function to the targeted column only. In Python:

from pyspark.sql.functions import UserDefinedFunction
from pyspark.sql.types import StringType

name="target_column"
udf = UserDefinedFunction(lambda x: 'new_value', StringType())
new_df = old_df.select(*[udf(column).alias(name) if column == name else column for column in old_df.columns])

new_df now has the same schema as old_df (assuming that old_df.target_column was of type StringType as well) but all values in column target_column will be new_value.

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. Spark Dataframe distinguish columns with duplicated name
  3. Pyspark: Split multiple array columns into rows
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to change a dataframe column from String type to Double type in PySpark?
  6. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  7. Retrieve top n in each group of a DataFrame in pyspark
  8. Dividing complex rows of dataframe to simple rows in Pyspark
  9. Filter Pyspark dataframe column with None value
  10. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  11. How to explode multiple columns of a dataframe in pyspark
  12. Spark add new column to dataframe with value from previous row
  13. PySpark: multiple conditions in when clause
  14. Pyspark: Replacing value in a column by searching a dictionary
  15. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  16. Create Spark DataFrame. Can not infer schema for type
  17. Pivot String column on Pyspark Dataframe
  18. Filtering DataFrame using the length of a column
  19. Multiple condition filter on dataframe
  20. How to return a “Tuple type” in a UDF in PySpark?
  21. How to perform union on two DataFrames with different amounts of columns in spark?
  22. Does spark predicate pushdown work with JDBC?
  23. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  24. Explode in PySpark
  25. Filtering a Pyspark DataFrame with SQL-like IN clause
  26. Passing a data frame column and external list to udf under withColumn
  27. Why is Apache-Spark – Python so slow locally as compared to pandas?
  28. PySpark create new column with mapping from a dict
  29. Using UDF ignores condition in when
  30. When to cache a DataFrame?

Leave a Comment