Algorithm to find articles with similar text

by

Edit distance isn’t a likely candidate, as it would be spelling/word-order dependent, and much more computationally expensive than Will is leading you to believe, considering the size and number of the documents you’d actually be interested in searching.

Something like Lucene is the way to go. You index all your documents, and then when you want to find documents similar to a given document, you turn your given document into a query, and search the index. Internally Lucene will be using tf-idf and an inverted index to make the whole process take an amount of time proportional to the number of documents that could possibly match, not the total number of documents in the collection.

More Related Contents:

  1. Ukkonen’s suffix tree algorithm in plain English
  2. Algorithm to convert an IEEE 754 double to a string?
  3. Generate list of all possible permutations of a string
  4. Difference between text and varchar (character varying)
  5. Upper vs Lower Case
  6. How many bytes does one Unicode character take?
  7. Split large string
  8. How to generate all the permutations of a multiset?
  9. Find the first un-repeated character in a string
  10. Finding the ranking of a word (permutations) with duplicate letters
  11. How to search a string in multiple files and return the names of files in Powershell?
  12. Best data structure for implementing a dictionary?
  13. Word comparison algorithm
  14. What is an easy way to tell if a list of words are anagrams of each other?
  15. What is tail recursion?
  16. Algorithm for finding the fewest rectangles to cover a set of rectangles without overlapping
  17. Getting the closest string match
  18. How do you detect Credit card type based on number?
  19. How to efficiently build a tree from a flat structure?
  20. How to concatenate strings with padding in sqlite
  21. The Sieve of Atkin
  22. How best to sum up lots of floating point numbers?
  23. Similar String algorithm
  24. Polygon enclosing a set of points
  25. Algorithm to find the most common substrings in a string
  26. When using the MOV mnemonic to load/copy a string to a memory register in MASM, are the characters stored in reverse order?
  27. Optimizing Conway’s ‘Game of Life’
  28. How can I replace newline characters using JSP and JSTL?
  29. Remove all line breaks at the beginning of a string in Swift
  30. finding common prefix of array of strings

Leave a Comment