nltk NaiveBayesClassifier training for sentiment analysis

by

You need to change your data structure. Here is your train list as it currently stands:

>>> train = [('I love this sandwich.', 'pos'),
('This is an amazing place!', 'pos'),
('I feel very good about these beers.', 'pos'),
('This is my best work.', 'pos'),
("What an awesome view", 'pos'),
('I do not like this restaurant', 'neg'),
('I am tired of this stuff.', 'neg'),
("I can't deal with this", 'neg'),
('He is my sworn enemy!', 'neg'),
('My boss is horrible.', 'neg')]

The problem is, though, that the first element of each tuple should be a dictionary of features. So I will change your list into a data structure that the classifier can work with:

>>> from nltk.tokenize import word_tokenize # or use some other tokenizer
>>> all_words = set(word.lower() for passage in train for word in word_tokenize(passage[0]))
>>> t = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train]

Your data should now be structured like this:

>>> t
[({'this': True, 'love': True, 'deal': False, 'tired': False, 'feel': False, 'is': False, 'am': False, 'an': False, 'sandwich': True, 'ca': False, 'best': False, '!': False, 'what': False, '.': True, 'amazing': False, 'horrible': False, 'sworn': False, 'awesome': False, 'do': False, 'good': False, 'very': False, 'boss': False, 'beers': False, 'not': False, 'with': False, 'he': False, 'enemy': False, 'about': False, 'like': False, 'restaurant': False, 'these': False, 'of': False, 'work': False, "n't": False, 'i': False, 'stuff': False, 'place': False, 'my': False, 'view': False}, 'pos'), . . .]

Note that the first element of each tuple is now a dictionary. Now that your data is in place and the first element of each tuple is a dictionary, you can train the classifier like so:

>>> import nltk
>>> classifier = nltk.NaiveBayesClassifier.train(t)
>>> classifier.show_most_informative_features()
Most Informative Features
                    this = True              neg : pos    =      2.3 : 1.0
                    this = False             pos : neg    =      1.8 : 1.0
                      an = False             neg : pos    =      1.6 : 1.0
                       . = True              pos : neg    =      1.4 : 1.0
                       . = False             neg : pos    =      1.4 : 1.0
                 awesome = False             neg : pos    =      1.2 : 1.0
                      of = False             pos : neg    =      1.2 : 1.0
                    feel = False             neg : pos    =      1.2 : 1.0
                   place = False             neg : pos    =      1.2 : 1.0
                horrible = False             pos : neg    =      1.2 : 1.0

If you want to use the classifier, you can do it like this. First, you begin with a test sentence:

>>> test_sentence = "This is the best band I've ever heard!"

Then, you tokenize the sentence and figure out which words the sentence shares with all_words. These constitute the sentence’s features.

>>> test_sent_features = {word: (word in word_tokenize(test_sentence.lower())) for word in all_words}

Your features will now look like this:

>>> test_sent_features
{'love': False, 'deal': False, 'tired': False, 'feel': False, 'is': True, 'am': False, 'an': False, 'sandwich': False, 'ca': False, 'best': True, '!': True, 'what': False, 'i': True, '.': False, 'amazing': False, 'horrible': False, 'sworn': False, 'awesome': False, 'do': False, 'good': False, 'very': False, 'boss': False, 'beers': False, 'not': False, 'with': False, 'he': False, 'enemy': False, 'about': False, 'like': False, 'restaurant': False, 'this': True, 'of': False, 'work': False, "n't": False, 'these': False, 'stuff': False, 'place': False, 'my': False, 'view': False}

Then you simply classify those features:

>>> classifier.classify(test_sent_features)
'pos' # note 'best' == True in the sentence features above

This test sentence appears to be positive.

More Related Contents:

  1. Classification using movie review corpus in NLTK/Python
  2. How to use Stanford Parser in NLTK using Python
  3. Creating a new corpus with NLTK
  4. Python NLTK pos_tag not returning the correct part-of-speech tag
  5. Stopword removal with NLTK
  6. Ordinal numbers replacement
  7. How to config nltk data directory from code?
  8. Convert words between verb/noun/adjective forms
  9. NLTK Named Entity recognition to a Python list
  10. Computing N Grams using Python
  11. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  12. How do I tokenize a string sentence in NLTK?
  13. How to get rid of punctuation using NLTK tokenizer?
  14. English grammar for parsing in NLTK
  15. NLTK and language detection
  16. Fast/Optimize N-gram implementations in python
  17. Extract Word from Synset using Wordnet in NLTK 3.0
  18. How do I do dependency parsing in NLTK?
  19. training data format for NLTK punkt
  20. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  21. Saving nltk drawn parse tree to image file
  22. Creating a custom categorized corpus in NLTK and Python
  23. tag generation from a text content
  24. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  25. What is NLTK POS tagger asking me to download?
  26. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  27. Fast n-gram calculation
  28. Using the Python NLTK (2.0b5) on the Google App Engine
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment