How to add a constant column in a Spark DataFrame?

by

Spark 2.2+

Spark 2.2 introduces typedLit to support Seq, Map, and Tuples (SPARK-19254) and following calls should be supported (Scala):

import org.apache.spark.sql.functions.typedLit

df.withColumn("some_array", typedLit(Seq(1, 2, 3)))
df.withColumn("some_struct", typedLit(("foo", 1, 0.3)))
df.withColumn("some_map", typedLit(Map("key1" -> 1, "key2" -> 2)))

Spark 1.3+ (lit), 1.4+ (array, struct), 2.0+ (map):

The second argument for DataFrame.withColumn should be a Column so you have to use a literal:

from pyspark.sql.functions import lit

df.withColumn('new_column', lit(10))

If you need complex columns you can build these using blocks like array:

from pyspark.sql.functions import array, create_map, struct

df.withColumn("some_array", array(lit(1), lit(2), lit(3)))
df.withColumn("some_struct", struct(lit("foo"), lit(1), lit(.3)))
df.withColumn("some_map", create_map(lit("key1"), lit(1), lit("key2"), lit(2)))

Exactly the same methods can be used in Scala.

import org.apache.spark.sql.functions.{array, lit, map, struct}

df.withColumn("new_column", lit(10))
df.withColumn("map", map(lit("key1"), lit(1), lit("key2"), lit(2)))

To provide names for structs use either alias on each field:

df.withColumn(
    "some_struct",
    struct(lit("foo").alias("x"), lit(1).alias("y"), lit(0.3).alias("z"))
 )

or cast on the whole object

df.withColumn(
    "some_struct", 
    struct(lit("foo"), lit(1), lit(0.3)).cast("struct<x: string, y: integer, z: double>")
 )

It is also possible, although slower, to use an UDF.

Note:

The same constructs can be used to pass constant arguments to UDFs or SQL functions.

More Related Contents:

  1. Spark Dataframe distinguish columns with duplicated name
  2. Pyspark: Split multiple array columns into rows
  3. How do I add a new column to a Spark DataFrame (using PySpark)?
  4. How to change a dataframe column from String type to Double type in PySpark?
  5. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  6. Retrieve top n in each group of a DataFrame in pyspark
  7. Updating a dataframe column in spark
  8. Dividing complex rows of dataframe to simple rows in Pyspark
  9. Filter Pyspark dataframe column with None value
  10. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  11. How to explode multiple columns of a dataframe in pyspark
  12. Spark add new column to dataframe with value from previous row
  13. PySpark: multiple conditions in when clause
  14. Pyspark: Replacing value in a column by searching a dictionary
  15. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  16. Create Spark DataFrame. Can not infer schema for type
  17. Pivot String column on Pyspark Dataframe
  18. Filtering DataFrame using the length of a column
  19. Multiple condition filter on dataframe
  20. How to return a “Tuple type” in a UDF in PySpark?
  21. Convert pyspark string to date format
  22. How to use JDBC source to write and read data in (Py)Spark?
  23. Applying UDFs on GroupedData in PySpark (with functioning python example)
  24. How to use a Scala class inside Pyspark
  25. Load CSV file with PySpark
  26. Cannot find col function in pyspark
  27. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  28. How do I convert an array (i.e. list) column to Vector
  29. Add column sum as new column in PySpark dataframe
  30. How to drop all columns with null values in a PySpark DataFrame?

Leave a Comment