How can I change column types in Spark SQL’s DataFrame?

by

Edit: Newest newest version

Since spark 2.x you should use dataset api instead when using Scala [1]. Check docs here:

https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Dataset.html#withColumn(colName:String,col:org.apache.spark.sql.Column):org.apache.spark.sql.DataFrame

If working with python, even though easier, I leave the link here as it’s a very highly voted question:

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html

>>> df.withColumn('age2', df.age + 2).collect()
[Row(age=2, name="Alice", age2=4), Row(age=5, name="Bob", age2=7)]

[1] https://spark.apache.org/docs/latest/sql-programming-guide.html:

In the Scala API, DataFrame is simply a type alias of Dataset[Row].
While, in Java API, users need to use Dataset to represent a
DataFrame.

Edit: Newest version

Since spark 2.x you can use .withColumn. Check the docs here:

https://spark.apache.org/docs/2.2.0/api/scala/index.html#org.apache.spark.sql.Dataset@withColumn(colName:String,col:org.apache.spark.sql.Column):org.apache.spark.sql.DataFrame

Oldest answer

Since Spark version 1.4 you can apply the cast method with DataType on the column:

import org.apache.spark.sql.types.IntegerType
val df2 = df.withColumn("yearTmp", df.year.cast(IntegerType))
    .drop("year")
    .withColumnRenamed("yearTmp", "year")

If you are using sql expressions you can also do:

val df2 = df.selectExpr("cast(year as int) year", 
                        "make", 
                        "model", 
                        "comment", 
                        "blank")

For more info check the docs:
http://spark.apache.org/docs/1.6.0/api/scala/#org.apache.spark.sql.DataFrame

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to convert rdd object to dataframe in spark
  3. How to define and use a User-Defined Aggregate Function in Spark SQL?
  4. How do I detect if a Spark DataFrame has a column
  5. How to define schema for custom type in Spark SQL?
  6. How to avoid duplicate columns after join?
  7. Flattening Rows in Spark
  8. Spark unionAll multiple dataframes
  9. How can I pass extra parameters to UDFs in Spark SQL?
  10. How to split a dataframe into dataframes with same column values?
  11. Provide schema while reading csv file as a dataframe
  12. How to zip two (or more) DataFrame in Spark
  13. How to flatmap a nested Dataframe in Spark
  14. How to define a custom aggregation function to sum a column of Vectors?
  15. Joining Spark dataframes on the key
  16. Spark DataFrame: does groupBy after orderBy maintain that order?
  17. Better way to convert a string field into timestamp in Spark
  18. Spark UDAF with ArrayType as bufferSchema performance issues
  19. Apache Spark how to append new column from list/array to Spark dataframe
  20. How to write unit tests in Spark 2.0+?
  21. how to filter out a null value from spark dataframe
  22. What are the various join types in Spark?
  23. Provide schema while reading csv file as a dataframe in Scala Spark
  24. Replace missing values with mean – Spark Dataframe
  25. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  26. How to calculate the size of dataframe in bytes in Spark?
  27. Exploding nested Struct in Spark dataframe
  28. Spark-Monotonically increasing id not working as expected in dataframe?
  29. Calculate Cosine Similarity Spark Dataframe
  30. Difference between two rows in Spark dataframe

Leave a Comment