Spark gives a StackOverflowError when training using ALS

by

The solution was to add checkpointing, which prevents the recursion used by the codebase from creating an overflow. First, create a new directory to store the checkpoints. Then, have your SparkContext use that directory for checkpointing. Here is the example in Python:

sc.setCheckpointDir('checkpoint/')

You may also need to add checkpointing to the ALS as well, but I haven’t been able to determine whether that makes a difference. To add a checkpoint there (probably not necessary), just do:

ALS.checkpointInterval = 2

More Related Contents:

  1. How to melt Spark DataFrame?
  2. Using a column value as a parameter to a spark DataFrame function
  3. Unpivot in spark-sql/pyspark
  4. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  5. Split Spark Dataframe string column into multiple columns
  6. Avoid performance impact of a single partition mode in Spark window functions
  7. How to check if spark dataframe is empty?
  8. How to split a list to multiple columns in Pyspark?
  9. pyspark dataframe filter or include based on list
  10. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  11. Adding a group count column to a PySpark dataframe
  12. Efficient pyspark join
  13. Multiple Spark applications with HiveContext
  14. How to loop through each row of dataFrame in pyspark
  15. Fill in null with previously known good value with pyspark
  16. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  17. PySpark – get row number for each row in a group
  18. Save ML model for future usage
  19. Pyspark: Pass multiple columns in UDF
  20. Keep only duplicates from a DataFrame regarding some field
  21. reading json file in pyspark
  22. How can I access python variable in Spark SQL?
  23. pyspark: rolling average using timeseries data
  24. How to exclude multiple columns in Spark dataframe in Python
  25. Stratified sampling with pyspark
  26. Get CSV to Spark dataframe
  27. How to turn off scientific notation in pyspark?
  28. Spark ALS predictAll returns empty
  29. Why is the fold action necessary in Spark?
  30. How to add a SparkListener from pySpark in Python?

Leave a Comment