How to save models from ML Pipeline to S3 or HDFS?

by

One way to save a model to HDFS is as following:

// persist model to HDFS
sc.parallelize(Seq(model), 1).saveAsObjectFile("hdfs:///user/root/linReg.model")

Saved model can then be loaded as:

val linRegModel = sc.objectFile[LinearRegressionModel]("linReg.model").first()

For more details see (ref)

More Related Contents:

  1. Matrix Multiplication in Apache Spark [closed]
  2. Resolving dependency problems in Apache Spark
  3. Add JAR files to a Spark job – spark-submit
  4. Spark: How to map Python with Scala or Java User Defined Functions?
  5. Spark read file from S3 using sc.textFile (“s3n://…)
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. How can I update a broadcast variable in spark streaming?
  8. MatchError while accessing vector column in Spark 2.0
  9. Apache Spark – foreach Vs foreachPartition When to use What?
  10. What are the Spark transformations that causes a Shuffle?
  11. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  12. Is gzip format supported in Spark?
  13. Change output filename prefix for DataFrame.write()
  14. How can I force Spark to execute code?
  15. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  16. How to vectorize DataFrame columns for ML algorithms?
  17. Spark ML VectorAssembler returns strange output
  18. Scala Iterable[(String, Long)] to SortedMap[String,Long] or TreeMap[String, Long] sorted in decreasing order of value [closed]
  19. How to flatten a struct in a Spark dataframe?
  20. How to handle categorical features with spark-ml?
  21. java.lang.ClassCastException using lambda expressions in spark job on remote server
  22. Is there an equivalent of Scala’s Either in Java 8?
  23. Logging while testing through Gradle
  24. Java 8 Stream, getting head and tail
  25. How do I resolve “WILL_NOT_PERFORM” MS AD reply when trying to change password in scala w/ the unboundid LDAP SDK?
  26. Using a Java library with Scala reserved words
  27. Addition of two RDD[mllib.linalg.Vector]’s
  28. Converting mysql table to spark dataset is very slow compared to same from csv file
  29. Which programming languages can be used to develop in Android? [duplicate]
  30. Why does SparkSession execute twice for one action?

Leave a Comment