About how to add a new column to an existing DataFrame with random values in Scala

by

Spark >= 2.3

It is possible to disable some optimizations using asNondeterministic method:

import org.apache.spark.sql.expressions.UserDefinedFunction

val f: UserDefinedFunction = ???
val fNonDeterministic: UserDefinedFunction = f.asNondeterministic

Please make sure you understand the guarantees before using this option.

Spark < 2.3

Function which is passed to udf should be deterministic (with possible exception of SPARK-20586) and nullary functions calls can be replaced by constants. If you want to generate random numbers use on of the built-in functions:

  • rand – Generate a random column with independent and identically distributed (i.i.d.) samples from U[0.0, 1.0].
  • randn – Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.

and transform the output to obtain required distribution for example:

(rand * Integer.MAX_VALUE).cast("bigint").cast("string")

More Related Contents:

  1. How to define and use a User-Defined Aggregate Function in Spark SQL?
  2. How can I pass extra parameters to UDFs in Spark SQL?
  3. SparkSQL: How to deal with null values in user defined function?
  4. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  5. Spark UDAF with ArrayType as bufferSchema performance issues
  6. Derive multiple columns from a single column in a Spark DataFrame
  7. How to convert rdd object to dataframe in spark
  8. Automatically and Elegantly flatten DataFrame in Spark SQL
  9. Spark extracting values from a Row
  10. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  11. Provide schema while reading csv file as a dataframe
  12. Processing multiple files as independent RDD’s in parallel
  13. How to flatmap a nested Dataframe in Spark
  14. MatchError while accessing vector column in Spark 2.0
  15. How to save DataFrame directly to Hive?
  16. Spark / Scala: forward fill with last observation
  17. Joining Spark dataframes on the key
  18. How to use Column.isin with list?
  19. How to write unit tests in Spark 2.0+?
  20. how to filter out a null value from spark dataframe
  21. How to compare two dataframe and print columns that are different in scala
  22. What are the various join types in Spark?
  23. Renaming column names of a DataFrame in Spark Scala
  24. What is going wrong with `unionAll` of Spark `DataFrame`?
  25. Exploding nested Struct in Spark dataframe
  26. How to get keys and values from MapType column in SparkSQL DataFrame
  27. Spark column string replace when present in other column (row)
  28. How to create a Dataset of Maps?
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment