Rename more than one column using withColumnRenamed

It is not possible to use a single withColumnRenamed call.

  • You can use DataFrame.toDF method*

    data.toDF('x3', 'x4')


    new_names = ['x3', 'x4']
  • It is also possible to rename with simple select:

    from pyspark.sql.functions import col
    mapping = dict(zip(['x1', 'x2'], ['x3', 'x4']))[col(c).alias(mapping.get(c, c)) for c in data.columns])

Similarly in Scala you can:

  • Rename all columns:

    val newNames = Seq("x3", "x4")
    data.toDF(newNames: _*)
  • Rename from mapping with select:

    val  mapping = Map("x1" -> "x3", "x2" -> "x4") => df(c).alias(mapping.get(c).getOrElse(c))): _*

    or foldLeft + withColumnRenamed

      case (data, (oldName, newName)) => data.withColumnRenamed(oldName, newName) 

* Not to be confused with RDD.toDF which is not a variadic functions, and takes column names as a list,

Leave a Comment