Is Spark’s KMeans unable to handle bigdata?

by

I think the ‘hanging’ is because your executors keep dying. As I mentioned in a side conversation, this code runs fine for me, locally and on a cluster, in Pyspark and Scala. However, it takes a lot longer than it should. It is almost all time spent in k-means|| initialization.

I opened https://issues.apache.org/jira/browse/SPARK-17389 to track two main improvements, one of which you can use now. Edit: really, see also https://issues.apache.org/jira/browse/SPARK-11560

First, there are some code optimizations that would speed up the init by about 13%.

However most of the issue is that it default to 5 steps of k-means|| init, when it seems that 2 is almost always just as good. You can set initialization steps to 2 to see a speedup, especially in the stage that’s hanging now.

In my (smaller) test on my laptop, init time went from 5:54 to 1:41 with both changes, mostly due to setting init steps.

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Encode and assemble multiple features in PySpark
  3. AttributeError: ‘DataFrame’ object has no attribute ‘map’
  4. How to create a custom Estimator in PySpark
  5. PySpark DataFrames – way to enumerate without converting to Pandas?
  6. Spark mllib predicting weird number or NaN
  7. python kmeans on string
  8. How to split Vector into columns – using PySpark
  9. How to create a large pandas dataframe from an sql query without running out of memory?
  10. Applying UDFs on GroupedData in PySpark (with functioning python example)
  11. creating spark data structure from multiline record
  12. Retrieve top n in each group of a DataFrame in pyspark
  13. How can PySpark be called in debug mode?
  14. Pyspark: explode json in column to multiple columns
  15. How to use a Scala class inside Pyspark
  16. Spark: Broadcast variables: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion
  17. Filter Pyspark dataframe column with None value
  18. Cast column containing multiple string date formats to DateTime in Spark
  19. How to explode multiple columns of a dataframe in pyspark
  20. Convert spark DataFrame column to python list
  21. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  22. Load CSV file with PySpark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. ImportError: No module named numpy on spark workers
  25. How to pass a constant value to Python UDF?
  26. How to zip two array columns in Spark SQL
  27. How to run stored procedure on SQL server from Spark (Databricks) JDBC python?
  28. When to cache a DataFrame?
  29. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  30. What is the difference between spark-submit and pyspark?

Leave a Comment