How to set timezone to UTC in Apache Spark?

by

Now you can use:

spark.conf.set("spark.sql.session.timeZone", "UTC")

Since https://issues.apache.org/jira/browse/SPARK-18936 in 2.2.0

Additionally, I set my default TimeZone to UTC to avoid implicit conversions

TimeZone.setDefault(TimeZone.getTimeZone("UTC"))

Otherwise you will get implicit conversions from your default Timezone to UTC when no Timezone information is present in the Timestamp you’re converting

Example:

val rawJson = """ {"some_date_field": "2018-09-14 16:05:37"} """

val dsRaw = sparkJob.spark.createDataset(Seq(rawJson))

val output =
  dsRaw
    .select(
      from_json(
        col("value"),
        new StructType(
          Array(
            StructField("some_date_field", DataTypes.TimestampType)
          )
        )
      ).as("parsed")
    ).select("parsed.*")

If my default TimeZone is Europe/Dublin which is GMT+1 and Spark sql session timezone is set to UTC, Spark will assume that “2018-09-14 16:05:37” is in Europe/Dublin TimeZone and do a conversion (result will be “2018-09-14 15:05:37”)

More Related Contents:

  1. How to flatten a struct in a Spark dataframe?
  2. How to handle this scenario in spark? [closed]
  3. How to do aggregate functions, may columns and extract back
  4. Spark: How to map Python with Scala or Java User Defined Functions?
  5. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  6. Spark sql how to explode without losing null values
  7. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  8. Spark Error – Unsupported class file major version
  9. Spark Strutured Streaming automatically converts timestamp to local time
  10. spark broadcast variable Map giving null value
  11. PySpark: java.lang.OutofMemoryError: Java heap space
  12. Change output filename prefix for DataFrame.write()
  13. How do I call a UDF on a Spark DataFrame using JAVA?
  14. Retain keys with null values while writing JSON in spark
  15. How to use Column.isin in Java?
  16. Why does SparkSession execute twice for one action?
  17. Running custom Java class in PySpark
  18. PermGen elimination in JDK 8
  19. Android Studio Gradle project “Unable to start the daemon process /initialization of VM”
  20. Java heap terminology: young, old and permanent generations?
  21. Retrieve top n in each group of a DataFrame in pyspark
  22. Find out what JVM Eclipse is running on
  23. Is there one JVM per Java application?
  24. Read Java JVM startup parameters (eg -Xmx)
  25. Maximum number of parameters in Java method declaration
  26. Differences between Just in Time compilation and On Stack Replacement
  27. Keep only duplicates from a DataFrame regarding some field
  28. If I compiled a Java file with the newest JDK, would an older JVM be able to run the .class files?
  29. Java for-loop optimization
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment