Spark Strutured Streaming automatically converts timestamp to local time

by

For me it worked to use:

spark.conf.set("spark.sql.session.timeZone", "UTC")

It tells the spark SQL to use UTC as a default timezone for timestamps. I used it in spark SQL for example:

select *, cast('2017-01-01 10:10:10' as timestamp) from someTable

I know it does not work in 2.0.1. but works in Spark 2.2. I used in SQLTransformer also and it worked.

I am not sure about streaming though.

More Related Contents:

  1. Change output filename prefix for DataFrame.write()
  2. How to handle this scenario in spark? [closed]
  3. How to do aggregate functions, may columns and extract back
  4. Resolving dependency problems in Apache Spark
  5. Add JAR files to a Spark job – spark-submit
  6. How to flatten a struct in a Spark dataframe?
  7. Spark: How to map Python with Scala or Java User Defined Functions?
  8. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  9. Spark read file from S3 using sc.textFile (“s3n://…)
  10. Spark sql how to explode without losing null values
  11. How to read records in JSON format from Kafka using Structured Streaming?
  12. How can I update a broadcast variable in spark streaming?
  13. Apache Spark – foreach Vs foreachPartition When to use What?
  14. Matrix Multiplication in Apache Spark [closed]
  15. spark broadcast variable Map giving null value
  16. What are the Spark transformations that causes a Shuffle?
  17. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  18. How to set timezone to UTC in Apache Spark?
  19. Is gzip format supported in Spark?
  20. How can I force Spark to execute code?
  21. How do I call a UDF on a Spark DataFrame using JAVA?
  22. Retain keys with null values while writing JSON in spark
  23. How to save models from ML Pipeline to S3 or HDFS?
  24. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  25. How to use Column.isin in Java?
  26. Why does SparkSession execute twice for one action?
  27. Jvm takes a long time to resolve ip-address for localhost
  28. Running Java gives “Error: could not open `C:\Program Files\Java\jre6\lib\amd64\jvm.cfg'”
  29. Why does SimpleDateFormat parse incorrect date?
  30. Unable to find velocity template resources

Leave a Comment