Convert date from String to Date format in Dataframes

by

Use to_date with Java SimpleDateFormat.

TO_DATE(CAST(UNIX_TIMESTAMP(date, 'MM/dd/yyyy') AS TIMESTAMP))

Example:

spark.sql("""
  SELECT TO_DATE(CAST(UNIX_TIMESTAMP('08/26/2016', 'MM/dd/yyyy') AS TIMESTAMP)) AS newdate"""
).show()

+----------+
|        dt|
+----------+
|2016-08-26|
+----------+

More Related Contents:

  1. Using a column value as a parameter to a spark DataFrame function
  2. Split Spark Dataframe string column into multiple columns
  3. Overwrite specific partitions in spark dataframe write method
  4. How to control partition size in Spark SQL
  5. Avoid performance impact of a single partition mode in Spark window functions
  6. How to import multiple csv files in a single load?
  7. How to check if spark dataframe is empty?
  8. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  9. Spark DataFrame: count distinct values of every column
  10. How to connect to remote hive server from spark [duplicate]
  11. pyspark dataframe filter or include based on list
  12. Use collect_list and collect_set in Spark SQL
  13. Why does format(“kafka”) fail with “Failed to find data source: kafka.” (even with uber-jar)?
  14. How to find mean of grouped Vector columns in Spark SQL?
  15. Generate a Spark StructType / Schema from a case class
  16. How to loop through each row of dataFrame in pyspark
  17. Fill in null with previously known good value with pyspark
  18. How to group by common element in array?
  19. How do I add an persistent column of row ids to Spark DataFrame?
  20. Convert null values to empty array in Spark DataFrame
  21. Keep only duplicates from a DataFrame regarding some field
  22. DataFrame partitionBy to a single Parquet file (per partition)
  23. Temp table caching with spark-sql
  24. Using windowing functions in Spark
  25. How can I access python variable in Spark SQL?
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. How to calculate Median in spark sqlContext for column of data type double
  28. Spark: disk I/O on stage boundaries explanation
  29. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment