What’s the difference between Spark ML and MLLIB packages

by

o.a.s.mllib contains old RDD-based API while o.a.s.ml contains new API build around Dataset and ML Pipelines. ml and mllib reached feature parity in 2.0.0 and mllib is slowly being deprecated (this already happened in case of linear regression) and most likely will be removed in the next major release.

So unless your goal is backward compatibility then the “right choice” is o.a.s.ml.

More Related Contents:

  1. How to handle categorical features with spark-ml?
  2. How to get word details from TF Vector RDD in Spark ML Lib?
  3. ALS model – how to generate full_u * v^t * v?
  4. Save ML model for future usage
  5. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  6. Finding duplicates from large data set using Apache Spark
  7. How to serve a Spark MLlib model?
  8. Encode and assemble multiple features in PySpark
  9. How to access element of a VectorUDT column in a Spark DataFrame?
  10. MatchError while accessing vector column in Spark 2.0
  11. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  12. How to find mean of grouped Vector columns in Spark SQL?
  13. How to create a custom Estimator in PySpark
  14. How to save models from ML Pipeline to S3 or HDFS?
  15. How to assign unique contiguous numbers to elements in a Spark RDD
  16. Spark ALS predictAll returns empty
  17. How to vectorize DataFrame columns for ML algorithms?
  18. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  19. Spark ML VectorAssembler returns strange output
  20. Using a column value as a parameter to a spark DataFrame function
  21. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  22. How to import multiple csv files in a single load?
  23. What is the maximum size for a broadcast object in Spark?
  24. How does createOrReplaceTempView work in Spark?
  25. PySpark – get row number for each row in a group
  26. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  27. Overwrite only some partitions in a partitioned spark Dataset
  28. Using windowing functions in Spark
  29. What is spark.driver.maxResultSize?
  30. How to get Kafka offsets for structured query for manual and reliable offset management?

Leave a Comment