spark.ml StringIndexer throws ‘Unseen label’ on fit()

by

Unseen label is a generic message which doesn’t correspond to a specific column. Most likely problem is with a following stage:

StringIndexer(inputCol="lang", outputCol="lang_idx")

with pl-PL present in train("lang") and not present in test("lang").

You can correct it using setHandleInvalid with skip:

from pyspark.ml.feature import StringIndexer

train = sc.parallelize([(1, "foo"), (2, "bar")]).toDF(["k", "v"])
test = sc.parallelize([(3, "foo"), (4, "foobar")]).toDF(["k", "v"])

indexer = StringIndexer(inputCol="v", outputCol="vi")
indexer.fit(train).transform(test).show()

## Py4JJavaError: An error occurred while calling o112.showString.
## : org.apache.spark.SparkException: Job aborted due to stage failure: 
##   ...
##   org.apache.spark.SparkException: Unseen label: foobar.

indexer.setHandleInvalid("skip").fit(train).transform(test).show()

## +---+---+---+
## |  k|  v| vi|
## +---+---+---+
## |  3|foo|1.0|
## +---+---+---+

or, in the latest versions, keep:

indexer.setHandleInvalid("keep").fit(train).transform(test).show()

## +---+------+---+
## |  k|     v| vi|
## +---+------+---+
## |  3|   foo|0.0|
## |  4|foobar|2.0|
## +---+------+---+

More Related Contents:

  1. How to access element of a VectorUDT column in a Spark DataFrame?
  2. How to loop through each row of dataFrame in pyspark
  3. Why does Spark think this is a cross / Cartesian join
  4. How to exclude multiple columns in Spark dataframe in Python
  5. How to pivot Spark DataFrame?
  6. How to melt Spark DataFrame?
  7. Spark SQL: apply aggregate functions to a list of columns
  8. Unpivot in spark-sql/pyspark
  9. DataFrame join optimization – Broadcast Hash Join
  10. Pyspark: Split multiple array columns into rows
  11. Avoid performance impact of a single partition mode in Spark window functions
  12. How to change a dataframe column from String type to Double type in PySpark?
  13. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  14. How to check if spark dataframe is empty?
  15. Filter Spark DataFrame based on another DataFrame that specifies denylist criteria
  16. pyspark dataframe filter or include based on list
  17. Filter Pyspark dataframe column with None value
  18. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  19. How to find mean of grouped Vector columns in Spark SQL?
  20. Fill in null with previously known good value with pyspark
  21. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  22. How do I add an persistent column of row ids to Spark DataFrame?
  23. Convert null values to empty array in Spark DataFrame
  24. Keep only duplicates from a DataFrame regarding some field
  25. How can I access python variable in Spark SQL?
  26. How do I convert an array (i.e. list) column to Vector
  27. Filtering DataFrame using the length of a column
  28. Stratified sampling with pyspark
  29. Why is predicate pushdown not used in typed Dataset API (vs untyped DataFrame API)?
  30. How to turn off scientific notation in pyspark?

Leave a Comment