Addition of two RDD[mllib.linalg.Vector]’s

by

This is actually a good question. I work with mllib regularly and did not realize these basic linear algebra operations are not easily accessible.

The point is that the underlying breeze vectors have all of the linear algebra manipulations you would expect – including of course basic element wise addition that you specifically mentioned.

However the breeze implementation is hidden from the outside world via:

[private mllib]

So then, from the outside world/public API perspective, how do we access those primitives?

Some of them are already exposed: e.g. sum of squares:

/**
 * Returns the squared distance between two Vectors.
 * @param v1 first Vector.
 * @param v2 second Vector.
 * @return squared distance between two Vectors.
 */
def sqdist(v1: Vector, v2: Vector): Double = { 
  ...
}

However the selection of such available methods is limited – and in fact does not include the basic operations including element wise addition, subtraction, multiplication, etc.

So here is the best I could see:

  • Convert the vectors to breeze:
  • Perform the vector operations in breeze
  • Convert back from breeze to mllib Vector

Here is some sample code:

val v1 = Vectors.dense(1.0, 2.0, 3.0)
val v2 = Vectors.dense(4.0, 5.0, 6.0)
val bv1 = new DenseVector(v1.toArray)
val bv2 = new DenseVector(v2.toArray)

val vectout = Vectors.dense((bv1 + bv2).toArray)
vectout: org.apache.spark.mllib.linalg.Vector = [5.0,7.0,9.0]

More Related Contents:

  1. MatchError while accessing vector column in Spark 2.0
  2. Calculate Cosine Similarity Spark Dataframe
  3. How to vectorize DataFrame columns for ML algorithms?
  4. Spark DataFrames when udf functions do not accept large enough input variables
  5. Spark ML VectorAssembler returns strange output
  6. Write single CSV file using spark-csv
  7. How to convert rdd object to dataframe in spark
  8. (Why) do we need to call cache or persist on a RDD
  9. Automatically and Elegantly flatten DataFrame in Spark SQL
  10. How do I skip a header from CSV files in Spark?
  11. Spark extracting values from a Row
  12. How to pass -D parameter or environment variable to Spark job?
  13. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  14. Append a column to Data Frame in Apache Spark 1.3
  15. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  16. Processing multiple files as independent RDD’s in parallel
  17. How to save DataFrame directly to Hive?
  18. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  19. Spark / Scala: forward fill with last observation
  20. How to use Column.isin with list?
  21. How to convert unix timestamp to date in Spark
  22. Apache Spark: Get number of records per partition
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. What is going wrong with `unionAll` of Spark `DataFrame`?
  26. How to process multi line input records in Spark
  27. How to sort by column in descending order in Spark SQL?
  28. How can I connect to a postgreSQL database into Apache Spark using scala?
  29. Spark column string replace when present in other column (row)
  30. Spark textFile vs wholeTextFiles

Leave a Comment