Fastest pairwise distance metric in python

by

Neither of the other answers quite answered the question – 1 was in Cython, one was slower. But both provided very useful hints. Following up on them suggests that scipy.spatial.distance.pdist is the way to go.

Here’s some code:

import numpy as np
import random
import sklearn.metrics.pairwise
import scipy.spatial.distance

r = np.array([random.randrange(1, 1000) for _ in range(0, 1000)])
c = r[:, None]

def option1(r):
    dists = np.abs(r - r[:, None])

def option2(r):
    dists = scipy.spatial.distance.pdist(r, 'cityblock')

def option3(r):
    dists = sklearn.metrics.pairwise.manhattan_distances(r)

Timing with IPython:

In [36]: timeit option1(r)
100 loops, best of 3: 5.31 ms per loop

In [37]: timeit option2(c)
1000 loops, best of 3: 1.84 ms per loop

In [38]: timeit option3(c)
100 loops, best of 3: 11.5 ms per loop

I didn’t try the Cython implementation (I can’t use it for this project), but comparing my results to the other answer that did, it looks like scipy.spatial.distance.pdist is roughly a third slower than the Cython implementation (taking into account the different machines by benchmarking on the np.abs solution).

More Related Contents:

  1. How to do n-D distance and nearest neighbor calculations on numpy arrays
  2. Sorting arrays in NumPy by column
  3. Computing the correlation coefficient between two multi-dimensional arrays
  4. Working with big data in python and numpy, not enough ram, how to save partial results on disc?
  5. ImportError in importing from sklearn: cannot import name check_build
  6. How to normalize a NumPy array to within a certain range?
  7. How to extend an array in-place in Numpy?
  8. Efficient distance calculation between N points and a reference in numpy/scipy
  9. Numpy: Divide each row by a vector element
  10. ValueError: Dimension mismatch
  11. Is there a NumPy function to return the first index of something in an array?
  12. Understanding NumPy’s einsum
  13. Intuition and idea behind reshaping 4D array to 2D array in NumPy
  14. Removing nan values from an array
  15. Reshape an array in NumPy
  16. binning data in python with scipy/numpy
  17. Calculating Pearson correlation and significance in Python
  18. How to split a numpy array in fixed size chunks with and without overlap?
  19. Improving FFT performance in Python
  20. Parameters to numpy’s fromfunction
  21. Creating a Confidence Ellipses in a sccatterplot using matplotlib
  22. Distance between point and a line (from two points)
  23. Why does numpy.linalg.solve() offer more precise matrix inversions than numpy.linalg.inv()?
  24. How to calculate cumulative normal distribution?
  25. load csv into 2D matrix with numpy for plotting
  26. How to display progress of scipy.optimize function?
  27. Fitting a Weibull distribution using Scipy
  28. How to use sklearn fit_transform with pandas and return dataframe instead of numpy array?
  29. Efficient dot products of large memory-mapped arrays
  30. sparse 3d matrix/array in Python?

Leave a Comment