Spark Equivalent of IF Then ELSE

Correct structure is either:

(when(col("iris_class") == 'Iris-setosa', 0)
.when(col("iris_class") == 'Iris-versicolor', 1)
.otherwise(2))

which is equivalent to

CASE 
    WHEN (iris_class="Iris-setosa") THEN 0
    WHEN (iris_class="Iris-versicolor") THEN 1 
    ELSE 2
END

or:

(when(col("iris_class") == 'Iris-setosa', 0)
    .otherwise(when(col("iris_class") == 'Iris-versicolor', 1)
        .otherwise(2)))

which is equivalent to:

CASE WHEN (iris_class="Iris-setosa") THEN 0 
     ELSE CASE WHEN (iris_class="Iris-versicolor") THEN 1 
               ELSE 2 
          END 
END

with general syntax:

when(condition, value).when(...)

or

when(condition, value).otherwise(...)

You probably mixed up things with Hive IF conditional:

IF(condition, if-true, if-false)

which can be used only in raw SQL with Hive support.

Leave a Comment