Calculate Cosine Similarity Spark Dataframe

by
  • DataFrame.rdd returns RDD[Row] not RDD[(T, U)]. You have to pattern match the Row or directly extract interesting parts.
  • ml Vector used with Datasets since Spark 2.0 is not the same as mllib Vector use by old API. You have to convert it to use with IndexedRowMatrix.
  • Index has to be Long not string.
import org.apache.spark.sql.Row

val irm = new IndexedRowMatrix(inClusters.rdd.map {
  Row(_, v: org.apache.spark.ml.linalg.Vector) => 
    org.apache.spark.mllib.linalg.Vectors.fromML(v)
}.zipWithIndex.map { case (v, i) => IndexedRow(i, v) })

More Related Contents:

  1. MatchError while accessing vector column in Spark 2.0
  2. Spark DataFrames when udf functions do not accept large enough input variables
  3. How to query JSON data column using Spark DataFrames?
  4. How do I detect if a Spark DataFrame has a column
  5. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  6. How to define schema for custom type in Spark SQL?
  7. How to avoid duplicate columns after join?
  8. Flattening Rows in Spark
  9. Spark unionAll multiple dataframes
  10. How can I pass extra parameters to UDFs in Spark SQL?
  11. How to split a dataframe into dataframes with same column values?
  12. Provide schema while reading csv file as a dataframe
  13. How to zip two (or more) DataFrame in Spark
  14. How to flatmap a nested Dataframe in Spark
  15. How to define a custom aggregation function to sum a column of Vectors?
  16. Joining Spark dataframes on the key
  17. Better way to convert a string field into timestamp in Spark
  18. Spark UDAF with ArrayType as bufferSchema performance issues
  19. How to write unit tests in Spark 2.0+?
  20. how to filter out a null value from spark dataframe
  21. What are the various join types in Spark?
  22. Provide schema while reading csv file as a dataframe in Scala Spark
  23. Replace missing values with mean – Spark Dataframe
  24. How to calculate the size of dataframe in bytes in Spark?
  25. Exploding nested Struct in Spark dataframe
  26. Spark-Monotonically increasing id not working as expected in dataframe?
  27. How to vectorize DataFrame columns for ML algorithms?
  28. How to create a Dataset of Maps?
  29. Filter spark DataFrame on string contains
  30. Spark ML VectorAssembler returns strange output

Leave a Comment