Python: tf-idf-cosine: to find document similarity

by

First off, if you want to extract count features and apply TF-IDF normalization and row-wise euclidean normalization you can do it in one operation with TfidfVectorizer:

>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> from sklearn.datasets import fetch_20newsgroups
>>> twenty = fetch_20newsgroups()

>>> tfidf = TfidfVectorizer().fit_transform(twenty.data)
>>> tfidf
<11314x130088 sparse matrix of type '<type 'numpy.float64'>'
    with 1787553 stored elements in Compressed Sparse Row format>

Now to find the cosine distances of one document (e.g. the first in the dataset) and all of the others you just need to compute the dot products of the first vector with all of the others as the tfidf vectors are already row-normalized.

As explained by Chris Clark in comments and here Cosine Similarity does not take into account the magnitude of the vectors. Row-normalised have a magnitude of 1 and so the Linear Kernel is sufficient to calculate the similarity values.

The scipy sparse matrix API is a bit weird (not as flexible as dense N-dimensional numpy arrays). To get the first vector you need to slice the matrix row-wise to get a submatrix with a single row:

>>> tfidf[0:1]
<1x130088 sparse matrix of type '<type 'numpy.float64'>'
    with 89 stored elements in Compressed Sparse Row format>

scikit-learn already provides pairwise metrics (a.k.a. kernels in machine learning parlance) that work for both dense and sparse representations of vector collections. In this case we need a dot product that is also known as the linear kernel:

>>> from sklearn.metrics.pairwise import linear_kernel
>>> cosine_similarities = linear_kernel(tfidf[0:1], tfidf).flatten()
>>> cosine_similarities
array([ 1.        ,  0.04405952,  0.11016969, ...,  0.04433602,
    0.04457106,  0.03293218])

Hence to find the top 5 related documents, we can use argsort and some negative array slicing (most related documents have highest cosine similarity values, hence at the end of the sorted indices array):

>>> related_docs_indices = cosine_similarities.argsort()[:-5:-1]
>>> related_docs_indices
array([    0,   958, 10576,  3277])
>>> cosine_similarities[related_docs_indices]
array([ 1.        ,  0.54967926,  0.32902194,  0.2825788 ])

The first result is a sanity check: we find the query document as the most similar document with a cosine similarity score of 1 which has the following text:

>>> print twenty.data[0]
From: [email protected] (where's my thing)
Subject: WHAT car is this!?
Nntp-Posting-Host: rac3.wam.umd.edu
Organization: University of Maryland, College Park
Lines: 15

 I was wondering if anyone out there could enlighten me on this car I saw
the other day. It was a 2-door sports car, looked to be from the late 60s/
early 70s. It was called a Bricklin. The doors were really small. In addition,
the front bumper was separate from the rest of the body. This is
all I know. If anyone can tellme a model name, engine specs, years
of production, where this car is made, history, or whatever info you
have on this funky looking car, please e-mail.

Thanks,
- IL
   ---- brought to you by your neighborhood Lerxst ----

The second most similar document is a reply that quotes the original message hence has many common words:

>>> print twenty.data[958]
From: [email protected] (Robert Seymour)
Subject: Re: WHAT car is this!?
Article-I.D.: reed.1993Apr21.032905.29286
Reply-To: [email protected]
Organization: Reed College, Portland, OR
Lines: 26

In article <[email protected]> [email protected] (where's my
thing) writes:
>
>  I was wondering if anyone out there could enlighten me on this car I saw
> the other day. It was a 2-door sports car, looked to be from the late 60s/
> early 70s. It was called a Bricklin. The doors were really small. In
addition,
> the front bumper was separate from the rest of the body. This is
> all I know. If anyone can tellme a model name, engine specs, years
> of production, where this car is made, history, or whatever info you
> have on this funky looking car, please e-mail.

Bricklins were manufactured in the 70s with engines from Ford. They are rather
odd looking with the encased front bumper. There aren't a lot of them around,
but Hemmings (Motor News) ususally has ten or so listed. Basically, they are a
performance Ford with new styling slapped on top.

>    ---- brought to you by your neighborhood Lerxst ----

Rush fan?

--
Robert Seymour              [email protected]
Physics and Philosophy, Reed College    (NeXTmail accepted)
Artificial Life Project         Reed College
Reed Solar Energy Project (SolTrain)    Portland, OR

More Related Contents:

  1. Python NLTK pos_tag not returning the correct part-of-speech tag
  2. Fast/Optimize N-gram implementations in python
  3. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  4. tag generation from a text content
  5. Save Naive Bayes Trained Classifier in NLTK
  6. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  7. How to get most informative features for scikit-learn classifier for different class?
  8. What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits?
  9. How do I download NLTK data?
  10. How to get the location of all text present in an image using OpenCV?
  11. How to get most informative features for scikit-learn classifiers?
  12. Extract upper or lower triangular part of a numpy matrix
  13. NLTK Named Entity recognition to a Python list
  14. Feature/Variable importance after a PCA analysis
  15. Does Any one got “AttributeError: ‘str’ object has no attribute ‘decode’ ” , while Loading a Keras Saved Model
  16. How do I tokenize a string sentence in NLTK?
  17. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  18. ValueError: Error when checking target: expected model_2 to have shape (None, 252, 252, 1) but got array with shape (300, 128, 128, 3)
  19. How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?
  20. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  21. Multivariate (polynomial) best fit curve in python?
  22. error installing nltk supporting packages : nltk.download()
  23. What is the role of TimeDistributed layer in Keras?
  24. Saving nltk drawn parse tree to image file
  25. Creating a custom categorized corpus in NLTK and Python
  26. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  27. Using Smote with Gridsearchcv in Scikit-learn
  28. Numpy Broadcast to perform euclidean distance vectorized
  29. How to use malt parser in python nltk
  30. ValueError: x and y must be the same size

Leave a Comment