What’s the fastest way in Python to calculate cosine similarity given sparse matrix data?

by

You can compute pairwise cosine similarity on the rows of a sparse matrix directly using sklearn. As of version 0.17 it also supports sparse output:

from sklearn.metrics.pairwise import cosine_similarity
from scipy import sparse

A =  np.array([[0, 1, 0, 0, 1], [0, 0, 1, 1, 1],[1, 1, 0, 1, 0]])
A_sparse = sparse.csr_matrix(A)

similarities = cosine_similarity(A_sparse)
print('pairwise dense output:\n {}\n'.format(similarities))

#also can output sparse matrices
similarities_sparse = cosine_similarity(A_sparse,dense_output=False)
print('pairwise sparse output:\n {}\n'.format(similarities_sparse))

Results:

pairwise dense output:
[[ 1.          0.40824829  0.40824829]
[ 0.40824829  1.          0.33333333]
[ 0.40824829  0.33333333  1.        ]]

pairwise sparse output:
(0, 1)  0.408248290464
(0, 2)  0.408248290464
(0, 0)  1.0
(1, 0)  0.408248290464
(1, 2)  0.333333333333
(1, 1)  1.0
(2, 1)  0.333333333333
(2, 0)  0.408248290464
(2, 2)  1.0

If you want column-wise cosine similarities simply transpose your input matrix beforehand: 

A_sparse.transpose()

More Related Contents:

  1. How do I make a function in python which takes a list of integers as an input and outputs smaller lists with only two values?
  2. Fast punctuation removal with pandas
  3. String concatenation of two pandas columns
  4. How to one-hot-encode from a pandas column containing a list?
  5. Calculate average of every x rows in a table and create new table
  6. Find column name in pandas that matches an array
  7. Efficiently return the index of the first value satisfying condition in array
  8. Conditionally fill column values based on another columns value in pandas
  9. Group duplicate column IDs in pandas dataframe
  10. Suppressing scientific notation in pandas? [duplicate]
  11. How to one hot encode variant length features?
  12. Using numpy.genfromtxt to read a csv file with strings containing commas
  13. pandas equivalent of np.where
  14. how to read certain columns from Excel using Pandas – Python
  15. which is faster for load: pickle or hdf5 in python
  16. Saving Matplotlib graphs to image as full screen
  17. Create new dataframe in pandas with dynamic names also add new column
  18. How can I count the number of consecutive TRUEs in a DataFrame?
  19. Pandas Latitude-Longitude to distance between successive rows [duplicate]
  20. pd.Timestamp versus np.datetime64: are they interchangeable for selected uses?
  21. Create new column based on values from other columns / apply a function of multiple columns, row-wise in Pandas
  22. Pandas in AWS lambda gives numpy error
  23. Pandas groupby with categories with redundant nan
  24. Filling in date gaps in MultiIndex Pandas Dataframe
  25. Different std in pandas vs numpy
  26. Python numpy: cannot convert datetime64[ns] to datetime64[D] (to use with Numba)
  27. Getting days since last occurence in Pandas DataFrame?
  28. How to use sklearn fit_transform with pandas and return dataframe instead of numpy array?
  29. mean from pandas and numpy differ
  30. Convert Pandas dataframe to Sparse Numpy Matrix directly

Leave a Comment