Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?

by

Spark Datasets require Encoders for data type which is about to be stored. For common types (atomics, product types) there is a number of predefined encoders available but you have to import these first from SparkSession.implicits to make it work:

val sparkSession: SparkSession = ???
import sparkSession.implicits._
val dataset = sparkSession.createDataset(dataList)

Alternatively you can provide directly an explicit 

import org.apache.spark.sql.{Encoder, Encoders}

val dataset = sparkSession.createDataset(dataList)(Encoders.product[SimpleTuple])

or implicit 

implicit val enc: Encoder[SimpleTuple] = Encoders.product[SimpleTuple]
val dataset = sparkSession.createDataset(dataList)

Encoder for the stored type.

Note that Encoders also provide a number of predefined Encoders for atomic types, and Encoders for complex ones, can derived with ExpressionEncoder.

Further reading:

More Related Contents:

  1. How to store custom objects in Dataset?
  2. Encoder error while trying to map dataframe row to updated row
  3. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  4. How to create a Dataset of Maps?
  5. Spark 2.0 Dataset vs DataFrame
  6. Perform a typed join in Scala with Spark Datasets
  7. How to get keys and values from MapType column in SparkSQL DataFrame
  8. Write single CSV file using spark-csv
  9. How to convert rdd object to dataframe in spark
  10. How to define and use a User-Defined Aggregate Function in Spark SQL?
  11. How to load local file in sc.textFile, instead of HDFS
  12. Stratified sampling in Spark
  13. How do I convert csv file to rdd
  14. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  15. How to flatmap a nested Dataframe in Spark
  16. Joining Spark dataframes on the key
  17. Modify collection inside a Spark RDD foreach
  18. How to write unit tests in Spark 2.0+?
  19. Customize SparkContext using sparkConf.set(..) when using spark-shell
  20. Replace missing values with mean – Spark Dataframe
  21. Spark: Transpose DataFrame Without Aggregating
  22. How to calculate the size of dataframe in bytes in Spark?
  23. Why does partition parameter of SparkContext.textFile not take effect?
  24. Exploding nested Struct in Spark dataframe
  25. How do I iterate RDD’s in apache spark (scala)
  26. How to compute cumulative sum using Spark
  27. Spark-Monotonically increasing id not working as expected in dataframe?
  28. Add Number of days column to Date Column in same dataframe for Spark Scala App
  29. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  30. How to create DataFrame from Scala’s List of Iterables?

Leave a Comment