Using my own corpus instead of movie_reviews corpus for Classification in NLTK

by

If you have you data in exactly the same structure as the movie_review corpus in NLTK, there are two ways to “hack” your way through:

1. Put your corpus directory into where you save the nltk.data

First check where is your nltk.data saved:

>>> import nltk
>>> nltk.data.find('corpora/movie_reviews')
FileSystemPathPointer(u'/home/alvas/nltk_data/corpora/movie_reviews')

Then move your directory to where the location where nltk_data/corpora is saved:

# Let's make a test corpus like `nltk.corpus.movie_reviews`
~$ mkdir my_movie_reviews
~$ mkdir my_movie_reviews/pos
~$ mkdir my_movie_reviews/neg
~$ echo "This is a great restaurant." > my_movie_reviews/pos/1.txt
~$ echo "Had a great time at chez jerome." > my_movie_reviews/pos/2.txt
~$ echo "Food fit for the ****" > my_movie_reviews/neg/1.txt
~$ echo "Slow service." > my_movie_reviews/neg/2.txt
~$ echo "README please" > my_movie_reviews/README
# Move it to `nltk_data/corpora/`
~$ mv my_movie_reviews/ nltk_data/corpora/

In your python code:

>>> import string
>>> from nltk.corpus import LazyCorpusLoader, CategorizedPlaintextCorpusReader
>>> from nltk.corpus import stopwords
>>> my_movie_reviews = LazyCorpusLoader('my_movie_reviews', CategorizedPlaintextCorpusReader, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
>>> mr = my_movie_reviews
>>>
>>> stop = stopwords.words('english')
>>> documents = [([w for w in mr.words(i) if w.lower() not in stop and w.lower() not in string.punctuation], i.split("https://stackoverflow.com/")[0]) for i in mr.fileids()]
>>> for i in documents:
...     print i
... 
([u'Food', u'fit', u'****'], u'neg')
([u'Slow', u'service'], u'neg')
([u'great', u'restaurant'], u'pos')
([u'great', u'time', u'chez', u'jerome'], u'pos')

(For more details, see https://github.com/nltk/nltk/blob/develop/nltk/corpus/util.py#L21 and https://github.com/nltk/nltk/blob/develop/nltk/corpus/init.py#L144)

2. Create your own CategorizedPlaintextCorpusReader

If you have no access to nltk.data directory and you want to use your own corpus, try this:

# Let's say that your corpus is saved on `/home/alvas/my_movie_reviews/`

>>> import string; from nltk.corpus import stopwords
>>> from nltk.corpus import CategorizedPlaintextCorpusReader
>>> mr = CategorizedPlaintextCorpusReader('/home/alvas/my_movie_reviews', r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
>>> stop = stopwords.words('english')
>>> documents = [([w for w in mr.words(i) if w.lower() not in stop and w.lower() not in string.punctuation], i.split("https://stackoverflow.com/")[0]) for i in mr.fileids()]
>>> 
>>> for doc in documents:
...     print doc
... 
([u'Food', u'fit', u'****'], 'neg')
([u'Slow', u'service'], 'neg')
([u'great', u'restaurant'], 'pos')
([u'great', u'time', u'chez', u'jerome'], 'pos')

Similar questions has been asked on Creating a custom categorized corpus in NLTK and Python and Using my own corpus for category classification in Python NLTK

Here’s the full code that will work:

import string
from itertools import chain

from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.classify import NaiveBayesClassifier as nbc
from nltk.corpus import CategorizedPlaintextCorpusReader
import nltk

mydir="/home/alvas/my_movie_reviews"

mr = CategorizedPlaintextCorpusReader(mydir, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
stop = stopwords.words('english')
documents = [([w for w in mr.words(i) if w.lower() not in stop and w.lower() not in string.punctuation], i.split("https://stackoverflow.com/")[0]) for i in mr.fileids()]

word_features = FreqDist(chain(*[i for i,j in documents]))
word_features = word_features.keys()[:100]

numtrain = int(len(documents) * 90 / 100)
train_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag in documents[:numtrain]]
test_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag  in documents[numtrain:]]

classifier = nbc.train(train_set)
print nltk.classify.accuracy(classifier, test_set)
classifier.show_most_informative_features(5)

More Related Contents:

  1. Creating a new corpus with NLTK
  2. Classification using movie review corpus in NLTK/Python
  3. Keep getting some permission denied error
  4. how to count average sentence length (in words) from a text file contains 100 sentences using python [closed]
  5. How can i calculate Caltalan numbers without a for loop?
  6. How to run independent transformations in parallel using PySpark?
  7. Installing Pygame on 64-bit Windows 7 and 64-bit Python 2.7
  8. Python NLTK pos_tag not returning the correct part-of-speech tag
  9. Stopword removal with NLTK
  10. Convert words between verb/noun/adjective forms
  11. Set scientific notation with fixed exponent and significant digits for multiple subplots
  12. cv2.imread always returns NoneType
  13. “This browser or app may not be secure” error while attempting to login in to Gmail account using GeckoDriver Firefox through Selenium and Python
  14. NLTK Named Entity recognition to a Python list
  15. How to control the download of files with Selenium + Python bindings in Chrome
  16. How do I tokenize a string sentence in NLTK?
  17. appcfg.py not working in command line
  18. training data format for NLTK punkt
  19. Pyspark filter dataframe by columns of another dataframe
  20. How to Navigate to a New Webpage In Selenium?
  21. Saving nltk drawn parse tree to image file
  22. Creating a custom categorized corpus in NLTK and Python
  23. Why doesn’t chained (interval) comparison work on numpy arrays?
  24. What is the difference between lemmatization vs stemming?
  25. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  26. downloading error using nltk.download()
  27. Quick implementation of character n-grams for word
  28. Using scipy curve_fit for a variable number of parameters
  29. SparkSQL on pyspark: how to generate time series?
  30. String formatting [str.format()] with a dictionary key which is a str() of a number

Leave a Comment