Spark ML VectorAssembler returns strange output

by

There is nothing strange about the output. Your vector seems to have lots of zero elements thus spark used it’s sparse representation.

To explain further :

It seems like your vector is composed of 18 elements (dimension).

This indices [0,1,6,9,14,17] from the vector contains non zero elements which are in order [17.0,15.0,3.0,1.0,4.0,2.0]

Sparse Vector representation is a way to save computational space thus easier and faster to compute. More on Sparse representation here.

Now of course you can convert that sparse representation to a dense representation but it comes at a cost.

In case you are interested in getting feature importance, thus I advise you to take a look at this.

More Related Contents:

  1. MatchError while accessing vector column in Spark 2.0
  2. How to vectorize DataFrame columns for ML algorithms?
  3. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  4. Dropping a nested column from Spark DataFrame
  5. How to define a custom aggregation function to sum a column of Vectors?
  6. How to save models from ML Pipeline to S3 or HDFS?
  7. Addition of two RDD[mllib.linalg.Vector]’s
  8. Calculate Cosine Similarity Spark Dataframe
  9. Spark DataFrames when udf functions do not accept large enough input variables
  10. (Why) do we need to call cache or persist on a RDD
  11. Automatically and Elegantly flatten DataFrame in Spark SQL
  12. How do I skip a header from CSV files in Spark?
  13. Spark extracting values from a Row
  14. How to pass -D parameter or environment variable to Spark job?
  15. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  16. Append a column to Data Frame in Apache Spark 1.3
  17. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  18. Processing multiple files as independent RDD’s in parallel
  19. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  20. Spark / Scala: forward fill with last observation
  21. How to convert unix timestamp to date in Spark
  22. Apache Spark: Get number of records per partition
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. How to process multi line input records in Spark
  26. How to sort by column in descending order in Spark SQL?
  27. Spark : how to run spark file from spark shell
  28. Spark Dataframe Random UUID changes after every transformation/action
  29. How to set hadoop configuration values from pyspark
  30. Spark textFile vs wholeTextFiles

Leave a Comment