Tokenize a paragraph into sentence and then into words in NLTK

by

You probably intended to loop over sent_text:

import nltk

sent_text = nltk.sent_tokenize(text) # this gives us a list of sentences
# now loop over each sentence and tokenize it separately
for sentence in sent_text:
    tokenized_text = nltk.word_tokenize(sentence)
    tagged = nltk.pos_tag(tokenized_text)
    print(tagged)

More Related Contents:

  1. Refering to a directory in a Flask app doesn’t work unless the path is absolute
  2. How to check if a word is an English word with Python?
  3. How to config nltk data directory from code?
  4. pip issue installing almost any library
  5. Classification using movie review corpus in NLTK/Python
  6. n-grams in python, four, five, six grams?
  7. Convert words between verb/noun/adjective forms
  8. nltk NaiveBayesClassifier training for sentiment analysis
  9. re.sub erroring with “Expected string or bytes-like object”
  10. All synonyms for word in python? [duplicate]
  11. Opening A large JSON file
  12. NLTK-based text processing with pandas
  13. nltk doesn’t add $NLTK_DATA to search path?
  14. How to get rid of punctuation using NLTK tokenizer?
  15. str.translate gives TypeError – Translate takes one argument (2 given), worked in Python 2
  16. Fast/Optimize N-gram implementations in python
  17. Counting the Frequency of words in a pandas data frame
  18. Extract list of Persons and Organizations using Stanford NER Tagger in NLTK
  19. training data format for NLTK punkt
  20. How to get all the hyponyms of a word/synset in python nltk and wordnet?
  21. Spell Checker for Python
  22. tag generation from a text content
  23. Save Naive Bayes Trained Classifier in NLTK
  24. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  25. Topic distribution: How do we see which document belong to which topic after doing LDA in python
  26. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  27. Fast n-gram calculation
  28. Using the Python NLTK (2.0b5) on the Google App Engine
  29. Getting 405 error while trying to download nltk data
  30. How to get most informative features for scikit-learn classifier for different class?

Leave a Comment