How to vectorize DataFrame columns for ML algorithms?

by

You can simply foldLeft over the Array of columns:

val transformed: DataFrame = df.columns.foldLeft(df)((df, arg) => str(arg, df))

Still, I will argue that it is not a good approach. Since src discards StringIndexerModel it cannot be used when you get new data. Because of that I would recommend using Pipeline:

import org.apache.spark.ml.Pipeline

val transformers: Array[org.apache.spark.ml.PipelineStage] = df.columns.map(
   cname => new StringIndexer()
     .setInputCol(cname)
     .setOutputCol(s"${cname}_index")
)

// Add the rest of your pipeline like VectorAssembler and algorithm
val stages: Array[org.apache.spark.ml.PipelineStage] = transformers ++ ???

val pipeline = new Pipeline().setStages(stages)
val model = pipeline.fit(df)
model.transform(df)

VectorAssembler can be included like this:

val assembler  = new VectorAssembler()
    .setInputCols(df.columns.map(cname => s"${cname}_index"))
    .setOutputCol("features")

val stages = transformers :+ assembler

You could also use RFormula, which is less customizable, but much more concise:

import org.apache.spark.ml.feature.RFormula

val rf = new RFormula().setFormula(" ~ uuid + url + browser - 1")
val rfModel = rf.fit(dataset)
rfModel.transform(dataset)

More Related Contents:

  1. MatchError while accessing vector column in Spark 2.0
  2. Spark ML VectorAssembler returns strange output
  3. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  4. Dropping a nested column from Spark DataFrame
  5. How to define a custom aggregation function to sum a column of Vectors?
  6. How to save models from ML Pipeline to S3 or HDFS?
  7. Addition of two RDD[mllib.linalg.Vector]’s
  8. Calculate Cosine Similarity Spark Dataframe
  9. Spark DataFrames when udf functions do not accept large enough input variables
  10. How to store custom objects in Dataset?
  11. Encode and assemble multiple features in PySpark
  12. Split 1 column into 3 columns in spark scala
  13. Provide schema while reading csv file as a dataframe
  14. How to flatmap a nested Dataframe in Spark
  15. Joining Spark dataframes on the key
  16. Modify collection inside a Spark RDD foreach
  17. How to write unit tests in Spark 2.0+?
  18. ALS model – how to generate full_u * v^t * v?
  19. Customize SparkContext using sparkConf.set(..) when using spark-shell
  20. Save ML model for future usage
  21. Spark: Transpose DataFrame Without Aggregating
  22. What’s the difference between Spark ML and MLLIB packages
  23. Exploding nested Struct in Spark dataframe
  24. How to compute cumulative sum using Spark
  25. Add Number of days column to Date Column in same dataframe for Spark Scala App
  26. Coalesce reduces parallelism of entire stage (spark)
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment