How to calculate the sentence similarity using word2vec model of gensim with python

by

This is actually a pretty challenging problem that you are asking. Computing sentence similarity requires building a grammatical model of the sentence, understanding equivalent structures (e.g. “he walked to the store yesterday” and “yesterday, he walked to the store”), finding similarity not just in the pronouns and verbs but also in the proper nouns, finding statistical co-occurences / relationships in lots of real textual examples, etc.

The simplest thing you could try — though I don’t know how well this would perform and it would certainly not give you the optimal results — would be to first remove all “stop” words (words like “the”, “an”, etc. that don’t add much meaning to the sentence) and then run word2vec on the words in both sentences, sum up the vectors in the one sentence, sum up the vectors in the other sentence, and then find the difference between the sums. By summing them up instead of doing a word-wise difference, you’ll at least not be subject to word order. That being said, this will fail in lots of ways and isn’t a good solution by any means (though good solutions to this problem almost always involve some amount of NLP, machine learning, and other cleverness).

So, short answer is, no, there’s no easy way to do this (at least not to do it well).

More Related Contents:

  1. Convert word2vec bin file to text
  2. LDA model generates different topics everytime i train on the same corpus
  3. Topic distribution: How do we see which document belong to which topic after doing LDA in python
  4. Pythonic way in loading multiple directories
  5. How do I read a large csv file with pandas?
  6. How to write the Fibonacci Sequence?
  7. Sieve of Eratosthenes – Finding Primes Python
  8. How to “test” NoneType in python?
  9. What is the best way to open a file for exclusive access in Python?
  10. List directory tree structure in python?
  11. ValueError: could not convert string to float: id
  12. Is it possible to get color gradients under curve in matplotlib?
  13. How to flatten a hierarchical index in columns
  14. What does [:] mean?
  15. Trace Python imports
  16. TensorFlow: Remember LSTM state for next batch (stateful LSTM)
  17. Pandas DataFrames with NaNs equality comparison
  18. LabelEncoder for categorical features?
  19. How to use pygame.KEYDOWN to execute something every time through a loop while the key is held down?
  20. How to exclude Optional unset values from a Pydantic model using FastAPI?
  21. Getting value from select tag using flask
  22. Is there an analysis speed or memory usage advantage to using HDF5 for large array storage (instead of flat binary files)?
  23. Cannot open new Jupyter Notebook [Permission Denied]
  24. Multiple keys per value
  25. How to install a package using the python-apt API
  26. 3-dimensional array in numpy
  27. What is the most efficient way to check if a value exists in a NumPy array?
  28. kill process with python
  29. How to smooth a curve for a dataset
  30. My plot in ipython does not show with pyplot.show()

Leave a Comment