How to create a Dataset of Maps?

by

It is not covered in 2.2, but can be easily addressed. You can add required Encoder using ExpressionEncoder, either explicitly:

import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder  
import org.apache.spark.sql.Encoder

spark
  .createDataset(Seq(Map(1 -> 2)))(ExpressionEncoder(): Encoder[Map[Int, Int]])

or implicitly:

implicit def mapIntIntEncoder: Encoder[Map[Int, Int]] = ExpressionEncoder()
spark.createDataset(Seq(Map(1 -> 2)))

More Related Contents:

  1. Encoder error while trying to map dataframe row to updated row
  2. How to store custom objects in Dataset?
  3. Spark 2.0 Dataset vs DataFrame
  4. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  5. Perform a typed join in Scala with Spark Datasets
  6. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  7. How to get keys and values from MapType column in SparkSQL DataFrame
  8. How to query JSON data column using Spark DataFrames?
  9. How do I detect if a Spark DataFrame has a column
  10. How to define schema for custom type in Spark SQL?
  11. How to avoid duplicate columns after join?
  12. Flattening Rows in Spark
  13. Spark unionAll multiple dataframes
  14. How can I pass extra parameters to UDFs in Spark SQL?
  15. How to split a dataframe into dataframes with same column values?
  16. Provide schema while reading csv file as a dataframe
  17. How to zip two (or more) DataFrame in Spark
  18. How to flatmap a nested Dataframe in Spark
  19. How to define a custom aggregation function to sum a column of Vectors?
  20. Joining Spark dataframes on the key
  21. Better way to convert a string field into timestamp in Spark
  22. How to write unit tests in Spark 2.0+?
  23. how to filter out a null value from spark dataframe
  24. What are the various join types in Spark?
  25. Provide schema while reading csv file as a dataframe in Scala Spark
  26. Replace missing values with mean – Spark Dataframe
  27. How to calculate the size of dataframe in bytes in Spark?
  28. Exploding nested Struct in Spark dataframe
  29. Spark-Monotonically increasing id not working as expected in dataframe?
  30. How to create DataFrame from Scala’s List of Iterables?

Leave a Comment