Pyspark changing type of column from date to string

by

Lets create some dummy data:

import datetime
from pyspark.sql import Row
from pyspark.sql.functions import col

row = Row("vacationdate")

df = sc.parallelize([
    row(datetime.date(2015, 10, 07)),
    row(datetime.date(1971, 01, 01))
]).toDF()

If you Spark >= 1.5.0 you can use date_format function:

from pyspark.sql.functions import date_format

(df
   .select(date_format(col("vacationdate"), "dd-MM-YYYY")
   .alias("date_string"))
   .show())

In Spark < 1.5.0 it can be done using Hive UDF:

df.registerTempTable("df")
sqlContext.sql(
    "SELECT date_format(vacationdate, 'dd-MM-YYYY') AS date_string FROM df")

It is of course still available in Spark >= 1.5.0.

If you don’t use HiveContext you can mimic date_format using UDF:

from pyspark.sql.functions import udf, lit
my_date_format = udf(lambda d, fmt: d.strftime(fmt))

df.select(
    my_date_format(col("vacationdate"), lit("%d-%m-%Y")).alias("date_string")
).show()

Please note it is using C standard format not a Java simple date format

More Related Contents:

  1. Convert pyspark string to date format
  2. How to use JDBC source to write and read data in (Py)Spark?
  3. Load CSV file with Spark
  4. Applying UDFs on GroupedData in PySpark (with functioning python example)
  5. Pyspark: Split multiple array columns into rows
  6. How do I add a new column to a Spark DataFrame (using PySpark)?
  7. How to perform union on two DataFrames with different amounts of columns in spark?
  8. How to change a dataframe column from String type to Double type in PySpark?
  9. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  10. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  11. Passing a data frame column and external list to udf under withColumn
  12. How to use a Scala class inside Pyspark
  13. Filter Pyspark dataframe column with None value
  14. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  15. Best way to get the max value in a Spark dataframe column
  16. Apache Spark Python Cosine Similarity over DataFrames
  17. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  18. Load CSV file with PySpark
  19. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  20. Cannot find col function in pyspark
  21. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  22. How to pass a constant value to Python UDF?
  23. How to zip two array columns in Spark SQL
  24. Concatenate two PySpark dataframes
  25. How do I convert an array (i.e. list) column to Vector
  26. Filtering DataFrame using the length of a column
  27. Add column sum as new column in PySpark dataframe
  28. How to add suffix and prefix to all columns in python/pyspark dataframe
  29. How to drop all columns with null values in a PySpark DataFrame?
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment