MatchError while accessing vector column in Spark 2.0

by

This has nothing to do with sparsity. Since Spark 2.0.0 ML Transformers no longer generate o.a.s.mllib.linalg.VectorUDT but o.a.s.ml.linalg.VectorUDT and are mapped locally to subclasses of o.a.s.ml.linalg.Vector. These are not compatible with old MLLib API which is moving towards deprecation in Spark 2.0.0.

You can convert between to “old” using Vectors.fromML:

import org.apache.spark.mllib.linalg.{Vectors => OldVectors}
import org.apache.spark.ml.linalg.{Vectors => NewVectors}

OldVectors.fromML(NewVectors.dense(1.0, 2.0, 3.0))
OldVectors.fromML(NewVectors.sparse(5, Seq(0 -> 1.0, 2 -> 2.0, 4 -> 3.0)))

but it make more sense to use ML implementation of LDA if you already use ML transformers.

For convenience you can use implicit conversions:

import scala.languageFeature.implicitConversions

object VectorConversions {
  import org.apache.spark.mllib.{linalg => mllib}
  import org.apache.spark.ml.{linalg => ml}

  implicit def toNewVector(v: mllib.Vector) = v.asML
  implicit def toOldVector(v: ml.Vector) = mllib.Vectors.fromML(v)
}

More Related Contents:

  1. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  2. Dropping a nested column from Spark DataFrame
  3. How to define a custom aggregation function to sum a column of Vectors?
  4. Calculate Cosine Similarity Spark Dataframe
  5. How to vectorize DataFrame columns for ML algorithms?
  6. Spark DataFrames when udf functions do not accept large enough input variables
  7. Spark ML VectorAssembler returns strange output
  8. How to convert rdd object to dataframe in spark
  9. How to define and use a User-Defined Aggregate Function in Spark SQL?
  10. Automatically and Elegantly flatten DataFrame in Spark SQL
  11. Spark extracting values from a Row
  12. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  13. Provide schema while reading csv file as a dataframe
  14. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  15. Processing multiple files as independent RDD’s in parallel
  16. How to save DataFrame directly to Hive?
  17. About how to add a new column to an existing DataFrame with random values in Scala
  18. Joining Spark dataframes on the key
  19. How to use Column.isin with list?
  20. How to write unit tests in Spark 2.0+?
  21. How to compare two dataframe and print columns that are different in scala
  22. What are the various join types in Spark?
  23. Renaming column names of a DataFrame in Spark Scala
  24. What is going wrong with `unionAll` of Spark `DataFrame`?
  25. Exploding nested Struct in Spark dataframe
  26. How to get keys and values from MapType column in SparkSQL DataFrame
  27. How to save models from ML Pipeline to S3 or HDFS?
  28. Addition of two RDD[mllib.linalg.Vector]’s
  29. How to create a Dataset of Maps?
  30. Filter spark DataFrame on string contains

Leave a Comment