Calculate cosine similarity given 2 sentence strings

by

A simple pure-Python implementation would be:

import math
import re
from collections import Counter

WORD = re.compile(r"\w+")


def get_cosine(vec1, vec2):
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])
    sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        return 0.0
    else:
        return float(numerator) / denominator


def text_to_vector(text):
    words = WORD.findall(text)
    return Counter(words)


text1 = "This is a foo bar sentence ."
text2 = "This sentence is similar to a foo bar sentence ."

vector1 = text_to_vector(text1)
vector2 = text_to_vector(text2)

cosine = get_cosine(vector1, vector2)

print("Cosine:", cosine)

Prints:

Cosine: 0.861640436855

The cosine formula used here is described here.

This does not include weighting of the words by tf-idf, but in order to use tf-idf, you need to have a reasonably large corpus from which to estimate tfidf weights.

You can also develop it further, by using a more sophisticated way to extract words from a piece of text, stem or lemmatise it, etc.

More Related Contents:

  1. What’s the fastest way in Python to calculate cosine similarity given sparse matrix data?
  2. Most efficient way to build single-element list from string with or without spaces? [closed]
  3. Python String concatenation with in double quotes [closed]
  4. Why does comparing strings using either ‘==’ or ‘is’ sometimes produce a different result?
  5. How to extract numbers from a string in Python?
  6. Convert hex string to int in Python
  7. String comparison in Python: is vs. == [duplicate]
  8. How can I fill out a Python string with spaces?
  9. Simpler way to create dictionary of separate variables?
  10. Stripping everything but alphanumeric chars from a string in Python
  11. How to display a float with two decimal places?
  12. How to convert a string of bytes into an int?
  13. Find common substring between two strings
  14. Removing numbers from string [closed]
  15. Extract a string between double quotes
  16. Python spacing and aligning strings
  17. Keeping only certain characters in a string using Python?
  18. What does a leading `\x` mean in a Python string `\xaa`
  19. Python reverse-stride slicing
  20. How to remove duplicates only if consecutive in a string? [duplicate]
  21. Iterate over the lines of a string
  22. How to convert a string of space- and comma- separated numbers into a list of int? [duplicate]
  23. Insert some string into given string at given index
  24. How to check whether a sentence is correct (simple grammar check in Python)?
  25. Fast/Optimize N-gram implementations in python
  26. Element-wise string concatenation in numpy
  27. How can I copy a Python string?
  28. Reverse a string without using reversed() or [::-1]?
  29. Are strings pooled in Python?
  30. Why do I get “TypeError: not all arguments converted during string formatting” trying to check for an even/odd number?

Leave a Comment