Creating a custom categorized corpus in NLTK and Python

by

Here is the answer to my question.
Since I was thinking about using two cases I think it’s good to cover both in case someone needs the answer in the future.
If you have the same setup as the movie_review corpus – several folders labeled in the same way you would like your labels to be called and containing the training data you can use this.

reader = CategorizedPlaintextCorpusReader('~/MainFolder/', r'.*\.txt', cat_pattern=r'(\w+)/*')

The other approach that I was considering is putting everything in a single folder and naming the files 0_neg.txt, 0_pos.txt, 1_neg.txt etc. The code for your reader should look something like:

reader = CategorizedPlaintextCorpusReader('~/MainFolder/', r'.*\.txt', cat_pattern=r'\d+_(\w+)\.txt')

I hope that this would help someone in the future.

More Related Contents:

  1. feature extraction in python for nlp
  2. How to use Stanford Parser in NLTK using Python
  3. Creating a new corpus with NLTK
  4. Python NLTK pos_tag not returning the correct part-of-speech tag
  5. Stopword removal with NLTK
  6. Ordinal numbers replacement
  7. How to config nltk data directory from code?
  8. Classification using movie review corpus in NLTK/Python
  9. Convert words between verb/noun/adjective forms
  10. re.sub erroring with “Expected string or bytes-like object”
  11. NLTK Named Entity recognition to a Python list
  12. Computing N Grams using Python
  13. Python – RegEx for splitting text into sentences (sentence-tokenizing) [duplicate]
  14. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  15. How do I tokenize a string sentence in NLTK?
  16. How to get rid of punctuation using NLTK tokenizer?
  17. English grammar for parsing in NLTK
  18. NLTK and language detection
  19. Fast/Optimize N-gram implementations in python
  20. Extract Word from Synset using Wordnet in NLTK 3.0
  21. How do I do dependency parsing in NLTK?
  22. training data format for NLTK punkt
  23. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  24. Saving nltk drawn parse tree to image file
  25. tag generation from a text content
  26. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  27. What is NLTK POS tagger asking me to download?
  28. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment