How to get rid of punctuation using NLTK tokenizer?

by

Take a look at the other tokenizing options that nltk provides here. For example, you can define a tokenizer that picks out sequences of alphanumeric characters as tokens and drops everything else:

from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+')
tokenizer.tokenize('Eighty-seven miles to go, yet.  Onward!')

Output:

['Eighty', 'seven', 'miles', 'to', 'go', 'yet', 'Onward']

More Related Contents:

  1. How do I tokenize a string sentence in NLTK?
  2. How to use Stanford Parser in NLTK using Python
  3. Creating a new corpus with NLTK
  4. Python NLTK pos_tag not returning the correct part-of-speech tag
  5. Stopword removal with NLTK
  6. Ordinal numbers replacement
  7. How to config nltk data directory from code?
  8. Classification using movie review corpus in NLTK/Python
  9. Convert words between verb/noun/adjective forms
  10. NLTK Named Entity recognition to a Python list
  11. Computing N Grams using Python
  12. Python – RegEx for splitting text into sentences (sentence-tokenizing) [duplicate]
  13. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  14. English grammar for parsing in NLTK
  15. NLTK and language detection
  16. How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How do I do dependency parsing in NLTK?
  20. training data format for NLTK punkt
  21. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  22. Saving nltk drawn parse tree to image file
  23. Creating a custom categorized corpus in NLTK and Python
  24. tag generation from a text content
  25. What is NLTK POS tagger asking me to download?
  26. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  27. Fast n-gram calculation
  28. Using the Python NLTK (2.0b5) on the Google App Engine
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment