Access Array column in Spark

by

ArrayType is represented in a Row as a scala.collection.mutable.WrappedArray. You can extract it using for example 

val arr: Seq[Double] = r.getAs[Seq[Double]]("x")

or

val i: Int = ???
val arr = r.getSeq[Double](i)

or even:

import scala.collection.mutable.WrappedArray

val arr: WrappedArray[Double] = r.getAs[WrappedArray[Double]]("x")

If DataFrame is relatively thin then pattern matching could be a better approach:

import org.apache.spark.sql.Row

df.rdd.map{case Row(x: Seq[Double]) => (x.toArray, x.sum)}

although you have to keep in mind that the type of the sequence is unchecked.

In Spark >= 1.6 you can also use Dataset as follows:

df.select("x").as[Seq[Double]].rdd

More Related Contents:

  1. Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)
  2. How to select the first row of each group?
  3. Querying Spark SQL DataFrame with complex types
  4. How to convert rdd object to dataframe in spark
  5. How to define and use a User-Defined Aggregate Function in Spark SQL?
  6. Automatically and Elegantly flatten DataFrame in Spark SQL
  7. Spark extracting values from a Row
  8. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  9. Provide schema while reading csv file as a dataframe
  10. Including null values in an Apache Spark Join
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. How to save DataFrame directly to Hive?
  15. Spark / Scala: forward fill with last observation
  16. About how to add a new column to an existing DataFrame with random values in Scala
  17. How to use Column.isin with list?
  18. How to pass Scala array into Scala vararg method?
  19. How to create and use a multi-dimensional array in Scala?
  20. How to compare two dataframe and print columns that are different in scala
  21. Apache Spark: Get number of records per partition
  22. Renaming column names of a DataFrame in Spark Scala
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. Why does `Array(0,1,2) == Array(0,1,2)` not return the expected result?
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to sort by column in descending order in Spark SQL?
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment