Spark mllib predicting weird number or NaN

by

The problem is that LinearRegressionWithSGD uses stochastic gradient descent (SGD) to optimize the weight vector of your linear model. SGD is really sensitive to the provided stepSize which is used to update the intermediate solution.

What SGD does is to calculate the gradient g of the cost function given a sample of the input points and the current weights w. In order to update the weights w you go for a certain distance in the opposite direction of g. The distance is your step size s.

w(i+1) = w(i) - s * g

Since you’re not providing an explicit step size value, MLlib assumes stepSize = 1. This seems to not work for your use case. I’d recommend you to try different step sizes, usually lower values, to see how LinearRegressionWithSGD behaves:

LinearRegressionWithSGD.train(parsedData, numIterartions = 10, stepSize = 0.001)

More Related Contents:

  1. Calling Java/Scala function from a task
  2. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  3. How to create a custom Estimator in PySpark
  4. How to find median and quantiles using Spark
  5. Spark Dataframe distinguish columns with duplicated name
  6. How to change dataframe column names in pyspark?
  7. I can’t seem to get –py-files on Spark to work
  8. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  9. Spark RDD to DataFrame python
  10. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  11. Why is Apache-Spark – Python so slow locally as compared to pandas?
  12. Spark groupByKey alternative
  13. How to extract an element from a array in pyspark
  14. PySpark create new column with mapping from a dict
  15. Tuning parameters for implicit pyspark.ml ALS matrix factorization model through pyspark.ml CrossValidator
  16. Spark iteration time increasing exponentially when using join
  17. Create Spark DataFrame. Can not infer schema for type
  18. PySpark DataFrames – way to enumerate without converting to Pandas?
  19. Getting Spark, Python, and MongoDB to work together
  20. How to transform data with sliding window over time series data in Pyspark
  21. Using UDF ignores condition in when
  22. Create single row dataframe from list of list PySpark
  23. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  24. pyspark parse fixed width text file
  25. Concatenating string by rows in pyspark
  26. What is the equivalent to scala.util.Try in pyspark?
  27. Filtering DataFrame using the length of a column
  28. How to process RDDs using a Python class?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment