How to access element of a VectorUDT column in a Spark DataFrame?

by

Convert output to float:

from pyspark.sql.types import DoubleType
from pyspark.sql.functions import lit, udf

def ith_(v, i):
    try:
        return float(v[i])
    except ValueError:
        return None

ith = udf(ith_, DoubleType())

Example usage:

from pyspark.ml.linalg import Vectors

df = sc.parallelize([
    (1, Vectors.dense([1, 2, 3])),
    (2, Vectors.sparse(3, [1], [9]))
]).toDF(["id", "features"])

df.select(ith("features", lit(1))).show()

## +-----------------+
## |ith_(features, 1)|
## +-----------------+
## |              2.0|
## |              9.0|
## +-----------------+

Explanation:

Output values have to be reserialized to equivalent Java objects. If you want to access values (beware of SparseVectors) you should use item method:

v.values.item(0)

which return standard Python scalars. Similarly if you want to access all values as a dense structure:

v.toArray().tolist()

More Related Contents:

  1. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  2. How to loop through each row of dataFrame in pyspark
  3. Why does Spark think this is a cross / Cartesian join
  4. How to exclude multiple columns in Spark dataframe in Python
  5. Finding duplicates from large data set using Apache Spark
  6. Find maximum row per group in Spark DataFrame
  7. How to add a constant column in a Spark DataFrame?
  8. Spark Dataframe distinguish columns with duplicated name
  9. How do I add a new column to a Spark DataFrame (using PySpark)?
  10. Retrieve top n in each group of a DataFrame in pyspark
  11. How to split a list to multiple columns in Pyspark?
  12. Dividing complex rows of dataframe to simple rows in Pyspark
  13. How to save/insert each DStream into a permanent table
  14. How to explode multiple columns of a dataframe in pyspark
  15. Spark load data and add filename as dataframe column
  16. Spark add new column to dataframe with value from previous row
  17. PySpark: how to resample frequencies
  18. PySpark: multiple conditions in when clause
  19. Pyspark: Replacing value in a column by searching a dictionary
  20. Create Spark DataFrame. Can not infer schema for type
  21. Pivot String column on Pyspark Dataframe
  22. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  23. spark dataframe drop duplicates and keep first
  24. pyspark: count distinct over a window
  25. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  26. How do I convert an array (i.e. list) column to Vector
  27. Filtering DataFrame using the length of a column
  28. Stratified sampling with pyspark
  29. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  30. How to turn off scientific notation in pyspark?

Leave a Comment