ALS model – how to generate full_u * v^t * v?

by

This new user is not the the matrix U, so you don’t have its latent representation in ‘k’ factors, you only know its full representation, i.e., all its ratings. full_u here means all of the new user ratings in a dense format (not the sparse format ratings are) e.g.:

[0 2 0 0 0 1 0] if user u has rated item 2 with a 2 and item 6 with a 1.

then you can get v pretty much like you did and turning it to a matrix in numpy for instance: 

pf = model.productFeatures()
Vt = np.matrix(np.asarray(pf.values().collect()))

then is is just a matter of multiplying:
full_u*Vt*Vt.T

Vt and V are transposed compared to the other answer but that’s just a matter of convention.

Note that the Vt*Vt.T product is fixed, so if you are going to use this for multiple new users it will be computationally more efficient to pre-compute it. Actually for more than one user it would be better to put all their ratings in bigU (in the same format as my one new user example) and just do the matrix product:
bigU*Vt*Vt.T to get all the ratings for all the new users. Might still be worth checking that the product is done in the most efficient way in terms of number of operations.

More Related Contents:

  1. How to handle categorical features with spark-ml?
  2. How to get word details from TF Vector RDD in Spark ML Lib?
  3. Save ML model for future usage
  4. What’s the difference between Spark ML and MLLIB packages
  5. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  6. Finding duplicates from large data set using Apache Spark
  7. How to serve a Spark MLlib model?
  8. Encode and assemble multiple features in PySpark
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. MatchError while accessing vector column in Spark 2.0
  11. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  12. How to find mean of grouped Vector columns in Spark SQL?
  13. How to create a custom Estimator in PySpark
  14. How to save models from ML Pipeline to S3 or HDFS?
  15. How to assign unique contiguous numbers to elements in a Spark RDD
  16. Spark ALS predictAll returns empty
  17. How to vectorize DataFrame columns for ML algorithms?
  18. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  19. Spark ML VectorAssembler returns strange output
  20. Spark Dataframe validating column names for parquet writes
  21. TypeError: Column is not iterable – How to iterate over ArrayType()?
  22. spark.sql.crossJoin.enabled for Spark 2.x
  23. Why do I have to explicitly tell Spark what to cache?
  24. multiple conditions for filter in spark data frames
  25. Why does sortBy transformation trigger a Spark job?
  26. Spark SQL broadcast hash join
  27. What is the difference between Apache Spark SQLContext vs HiveContext?
  28. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  29. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  30. What is spark.driver.maxResultSize?

Leave a Comment