tag generation from a text content

by

One way to do this would be to extract words that occur more frequently in a document than you would expect them to by chance. For example, say in a larger collection of documents the term ‘Markov’ is almost never seen. However, in a particular document from the same collection Markov shows up very frequently. This would suggest that Markov might be a good keyword or tag to associate with the document.

To identify keywords like this, you could use the point-wise mutual information of the keyword and the document. This is given by PMI(term, doc) = log [ P(term, doc) / (P(term)*P(doc)) ]. This will roughly tell you how much less (or more) surprised you are to come across the term in the specific document as appose to coming across it in the larger collection.

To identify the 5 best keywords to associate with a document, you would just sort the terms by their PMI score with the document and pick the 5 with the highest score.

If you want to extract multiword tags, see the StackOverflow question How to extract common / significant phrases from a series of text entries.

Borrowing from my answer to that question, the NLTK collocations how-to covers how to do
extract interesting multiword expressions using n-gram PMI in a about 7 lines of code, e.g.:

import nltk
from nltk.collocations import *
bigram_measures = nltk.collocations.BigramAssocMeasures()

# change this to read in your data
finder = BigramCollocationFinder.from_words(
   nltk.corpus.genesis.words('english-web.txt'))

# only bigrams that appear 3+ times
finder.apply_freq_filter(3) 

# return the 5 n-grams with the highest PMI
finder.nbest(bigram_measures.pmi, 5)

More Related Contents:

  1. Python NLTK pos_tag not returning the correct part-of-speech tag
  2. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  3. Extract email id from text file by showing path
  4. How to use Stanford Parser in NLTK using Python
  5. Creating a new corpus with NLTK
  6. Ordinal numbers replacement
  7. How to config nltk data directory from code?
  8. Classification using movie review corpus in NLTK/Python
  9. nltk NaiveBayesClassifier training for sentiment analysis
  10. NLTK Named Entity recognition to a Python list
  11. Computing N Grams using Python
  12. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  13. How do I tokenize a string sentence in NLTK?
  14. How to get rid of punctuation using NLTK tokenizer?
  15. English grammar for parsing in NLTK
  16. NLTK and language detection
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How do I do dependency parsing in NLTK?
  20. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  21. Saving nltk drawn parse tree to image file
  22. Creating a custom categorized corpus in NLTK and Python
  23. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  24. Save Naive Bayes Trained Classifier in NLTK
  25. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  26. What is NLTK POS tagger asking me to download?
  27. Fast n-gram calculation
  28. Using the Python NLTK (2.0b5) on the Google App Engine
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment