Encode an ADT / sealed trait hierarchy into Spark DataSet column

by

TL;DR There is no good solution right now, and given Spark SQL / Dataset implementation, it is unlikely there will be one in the foreseeable future.

You can use generic kryo or java encoder

val occupation: Seq[Occupation] = Seq(SoftwareEngineer, Wizard(1), Other("foo"))
spark.createDataset(occupation)(org.apache.spark.sql.Encoders.kryo[Occupation])

but is hardly useful in practice.

UDT API provides another possible approach as for now (Spark 1.6, 2.0, 2.1-SNAPSHOT) it is private and requires quite a lot boilerplate code (you can check o.a.s.ml.linalg.VectorUDT to see example implementation).

More Related Contents:

  1. How to store custom objects in Dataset?
  2. Encoder error while trying to map dataframe row to updated row
  3. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  4. How to create a Dataset of Maps?
  5. Spark 2.0 Dataset vs DataFrame
  6. Perform a typed join in Scala with Spark Datasets
  7. How to get keys and values from MapType column in SparkSQL DataFrame
  8. Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only
  9. How does HashPartitioner work?
  10. Spark – load CSV file as DataFrame?
  11. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  12. How to define partitioning of DataFrame?
  13. Case class equality in Apache Spark
  14. Dropping a nested column from Spark DataFrame
  15. Spark dataframe write method writing many small files
  16. Filling gaps in timeseries Spark
  17. Reading DataFrame from partitioned parquet file
  18. java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags
  19. How to use COGROUP for large datasets
  20. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  21. Parsing multiline records in Scala
  22. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  23. Create new Dataframe with empty/null field values
  24. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  25. DataFrame-ified zipWithIndex
  26. How to explode an array into multiple columns in Spark
  27. Why is join not possible after show operator?
  28. How can I connect to a postgreSQL database into Apache Spark using scala?
  29. Spark column string replace when present in other column (row)
  30. How to vectorize DataFrame columns for ML algorithms?

Leave a Comment