How do I tokenize a string sentence in NLTK?

by

This is actually on the main page of nltk.org:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

More Related Contents:

  1. How to get rid of punctuation using NLTK tokenizer?
  2. How to use Stanford Parser in NLTK using Python
  3. Creating a new corpus with NLTK
  4. Python NLTK pos_tag not returning the correct part-of-speech tag
  5. Stopword removal with NLTK
  6. Ordinal numbers replacement
  7. How to config nltk data directory from code?
  8. Classification using movie review corpus in NLTK/Python
  9. Convert words between verb/noun/adjective forms
  10. NLTK Named Entity recognition to a Python list
  11. Computing N Grams using Python
  12. Python – RegEx for splitting text into sentences (sentence-tokenizing) [duplicate]
  13. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  14. English grammar for parsing in NLTK
  15. NLTK and language detection
  16. How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How do I do dependency parsing in NLTK?
  20. training data format for NLTK punkt
  21. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  22. Saving nltk drawn parse tree to image file
  23. Creating a custom categorized corpus in NLTK and Python
  24. tag generation from a text content
  25. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  26. What is NLTK POS tagger asking me to download?
  27. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  28. Fast n-gram calculation
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment