Spark Dataframe Random UUID changes after every transformation/action

by

It is an expected behavior. User defined functions have to be deterministic:

The user-defined functions must be deterministic. Due to optimization,
duplicate invocations may be eliminated or the function may even be
invoked more times than it is present in the query.

If you want to include non-deterministic function and preserve the output you should write intermediate data to a persistent storage and read it back. Checkpointing or caching may work in some simple cases but it won’t be reliable in general.

If upstream process is deterministic (for starters there is shuffle) you could try to use rand function with seed, convert to byte array and pass to UUID.nameUUIDFromBytes.

See also: About how to add a new column to an existing DataFrame with random values in Scala

Note: SPARK-20586 introduced deterministic flag, which can disable certain optimization, but it is not clear how it behaves when data is persisted and a loss of executor occurs.

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Append a column to Data Frame in Apache Spark 1.3
  9. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  10. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  11. How to zip two (or more) DataFrame in Spark
  12. Joining Spark dataframes on the key
  13. Apache Spark how to append new column from list/array to Spark dataframe
  14. Create new Dataframe with empty/null field values
  15. Provide schema while reading csv file as a dataframe in Scala Spark
  16. Renaming column names of a DataFrame in Spark Scala
  17. Derive multiple columns from a single column in a Spark DataFrame
  18. Replace missing values with mean – Spark Dataframe
  19. What is going wrong with `unionAll` of Spark `DataFrame`?
  20. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  21. Replace null values in Spark DataFrame
  22. How to get keys and values from MapType column in SparkSQL DataFrame
  23. Add Number of days column to Date Column in same dataframe for Spark Scala App
  24. Spark DataFrames when udf functions do not accept large enough input variables
  25. Filter spark DataFrame on string contains
  26. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  27. How to create DataFrame from Scala’s List of Iterables?
  28. Scala-Spark Dynamically call groupby and agg with parameter values
  29. Joining two dataframes without a common column
  30. Difference between === null and isNull in Spark DataDrame

Leave a Comment