Spark MLlib LDA, how to infer the topics distribution of a new unseen document?

by

As of Spark 1.5 this functionality has not been implemented for the DistributedLDAModel. What you’re going to need to do is convert your model to a LocalLDAModel using the toLocal method and then call the topicDistributions(documents: RDD[(Long, Vector]) method where documents are the new (i.e. out-of-training) documents, something like this:

newDocuments: RDD[(Long, Vector)] = ...
val topicDistributions = distLDA.toLocal.topicDistributions(newDocuments)

This is going to be less accurate than the EM algorithm that this paper suggests, but it will work. Alternatively, you could just use the new online variational EM training algorithm which already results in a LocalLDAModel. In addition to being faster, this new algorithm is also preferable due to the fact that it, unlike the older EM algorithm for fitting DistributedLDAModels, is optimizing the parameters (alphas) of the Dirichlet prior over the topic mixing weights for the documents. According to Wallach, et. al., optimization of the alphas is pretty important for obtaining good topics.

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. How to serve a Spark MLlib model?
  3. How to handle categorical features with spark-ml?
  4. How to get word details from TF Vector RDD in Spark ML Lib?
  5. ALS model – how to generate full_u * v^t * v?
  6. Save ML model for future usage
  7. What’s the difference between Spark ML and MLLIB packages
  8. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  9. How to assign unique contiguous numbers to elements in a Spark RDD
  10. Spark ALS predictAll returns empty
  11. Using a column value as a parameter to a spark DataFrame function
  12. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  13. Split Spark Dataframe string column into multiple columns
  14. What does “Stage Skipped” mean in Apache Spark web UI?
  15. How to control partition size in Spark SQL
  16. How are stages split into tasks in Spark?
  17. How to tune spark executor number, cores and executor memory?
  18. How to split a list to multiple columns in Pyspark?
  19. Matrix Multiplication in Apache Spark [closed]
  20. Apache Spark does not delete temporary directories
  21. How to find mean of grouped Vector columns in Spark SQL?
  22. Generate a Spark StructType / Schema from a case class
  23. Fill in null with previously known good value with pyspark
  24. Which cluster type should I choose for Spark? [closed]
  25. Overwrite only some partitions in a partitioned spark Dataset
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. How to calculate Median in spark sqlContext for column of data type double
  28. How to vectorize DataFrame columns for ML algorithms?
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Understanding Spark’s caching

Leave a Comment