How to get word details from TF Vector RDD in Spark ML Lib?

by

Well, you can’t. Since hashing is non-injective there is no inverse function. In other words infinite number of tokens can map to a single bucket so it is impossible to tell which one is actually there.

If you’re using a large hash and number of unique tokens is relatively low then you can try to create a lookup table from bucket to possible tokens from your dataset. It is one-to-many mapping but if above conditions are met number of conflicts should be relatively low.

If you need a reversible transformation you can use combine Tokenizer and StringIndexer and build a sparse feature vector manually.

See also: What hashing function does Spark use for HashingTF and how do I duplicate it?

Edit:

In Spark 1.5+ (PySpark 1.6+) you can use CountVectorizer which applies reversible transformation and stores vocabulary.

Python:

from pyspark.ml.feature import CountVectorizer

df = sc.parallelize([
    (1, ["foo", "bar"]), (2, ["foo", "foobar", "baz"])
]).toDF(["id", "tokens"])

vectorizer = CountVectorizer(inputCol="tokens", outputCol="features").fit(df)
vectorizer.vocabulary
## ('foo', 'baz', 'bar', 'foobar')

Scala:

import org.apache.spark.ml.feature.{CountVectorizer, CountVectorizerModel}

val df = sc.parallelize(Seq(
    (1, Seq("foo", "bar")), (2, Seq("foo", "foobar", "baz"))
)).toDF("id", "tokens")

val model: CountVectorizerModel = new CountVectorizer()
  .setInputCol("tokens")
  .setOutputCol("features")
  .fit(df)

model.vocabulary
// Array[String] = Array(foo, baz, bar, foobar)

where element at the 0th position corresponds to index 0, element at the 1st position to index 1 and so on.

More Related Contents:

  1. How to handle categorical features with spark-ml?
  2. ALS model – how to generate full_u * v^t * v?
  3. Save ML model for future usage
  4. What’s the difference between Spark ML and MLLIB packages
  5. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  6. Finding duplicates from large data set using Apache Spark
  7. How to serve a Spark MLlib model?
  8. Encode and assemble multiple features in PySpark
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. MatchError while accessing vector column in Spark 2.0
  11. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  12. How to find mean of grouped Vector columns in Spark SQL?
  13. How to create a custom Estimator in PySpark
  14. How to save models from ML Pipeline to S3 or HDFS?
  15. How to assign unique contiguous numbers to elements in a Spark RDD
  16. Spark ALS predictAll returns empty
  17. How to vectorize DataFrame columns for ML algorithms?
  18. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  19. Spark ML VectorAssembler returns strange output
  20. What is the difference between map and flatMap and a good use case for each?
  21. Create a custom Transformer in PySpark ML
  22. Best Practice to launch Spark Applications via Web Application?
  23. How to save/insert each DStream into a permanent table
  24. How DAG works under the covers in RDD?
  25. Mind blown: RDD.zip() method
  26. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  27. Addition of two RDD[mllib.linalg.Vector]’s
  28. How to exclude multiple columns in Spark dataframe in Python
  29. Why does Spark job fail with “too many open files”?
  30. Stratified sampling with pyspark

Leave a Comment