Try this k-means variation: Initialization: choose k centers from the dataset at random, or even better using kmeans++ strategy for each point, compute the distance to its nearest cluster center, and build a heap for this draw points from the heap, and assign them to the nearest cluster, unless the cluster is already overfull. If … Read more
The base idea is to evaluate cluster scoring on sample data, usally it is distance inside cluster and distance between clusters. The more this measure the better clustering, based on this mesure you can select best clustring paramters. One of metrics can be found here http://alias-i.com/lingpipe/docs/api/com/aliasi/cluster/ClusterScore.html
This might do the trick: apply Lloyd’s algorithm to get k centroids. Sort the centroids by descending size of their associated clusters in an array. For i = 1 through k-1, push the data points in cluster i with minimal distance to any other centroid j (i < j ≤ k) off to j and … Read more
I think the ‘hanging’ is because your executors keep dying. As I mentioned in a side conversation, this code runs fine for me, locally and on a cluster, in Pyspark and Scala. However, it takes a lot longer than it should. It is almost all time spent in k-means|| initialization. I opened https://issues.apache.org/jira/browse/SPARK-17389 to track … Read more
Univariate k-means clustering can be solved in O(kn) time (on already sorted input) based on theoretical results on Monge matrices, but the approach was not popular most likely due to numerical instability and also perhaps coding challenges. A better option is an O(knlgn) method that is now implemented in Ckmeans.1d.dp version 3.4.6. This implementation is … Read more
This is a simpler example: from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.cluster import KMeans from sklearn.metrics import adjusted_rand_score documents = [“Human machine interface for lab abc computer applications”, “A survey of user opinion of computer system response time”, “The EPS user interface management system”, “System and human system engineering testing of EPS”, “Relation of user perceived … Read more
The official Java Sound Programmer Guide walks through reading and writing audio files. This article by A Greensted: Reading and Writing Wav Files in java should be helpful. The WavFile class is very useful and it can be tweaked to return the entire data array instead of buffered fragments.
Here’s a small kmeans that uses any of the 20-odd distances in scipy.spatial.distance, or a user function. Comments would be welcome (this has had only one user so far, not enough); in particular, what are your N, dim, k, metric ? #!/usr/bin/env python # kmeans.py using any of the 20-odd metrics in scipy.spatial.distance # kmeanssample … Read more
You can maximize the Bayesian Information Criterion (BIC): BIC(C | X) = L(X | C) – (p / 2) * log n where L(X | C) is the log-likelihood of the dataset X according to model C, p is the number of parameters in the model C, and n is the number of points in … Read more