How to tweak the NLTK sentence tokenizer

by

You need to supply a list of abbreviations to the tokenizer, like so:

from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
punkt_param = PunktParameters()
punkt_param.abbrev_types = set(['dr', 'vs', 'mr', 'mrs', 'prof', 'inc'])
sentence_splitter = PunktSentenceTokenizer(punkt_param)
text = "is THAT what you mean, Mrs. Hussey?"
sentences = sentence_splitter.tokenize(text)

sentences is now:

['is THAT what you mean, Mrs. Hussey?']

Update: This does not work if the last word of the sentence has an apostrophe or a quotation mark attached to it (like Hussey?’). So a quick-and-dirty way around this is to put spaces in front of apostrophes and quotes that follow sentence-end symbols (.!?):

text = text.replace('?"', '? "').replace('!"', '! "').replace('."', '. "')

More Related Contents:

  1. How to use Stanford Parser in NLTK using Python
  2. Creating a new corpus with NLTK
  3. Python NLTK pos_tag not returning the correct part-of-speech tag
  4. Stopword removal with NLTK
  5. Ordinal numbers replacement
  6. How to config nltk data directory from code?
  7. Classification using movie review corpus in NLTK/Python
  8. Convert words between verb/noun/adjective forms
  9. nltk NaiveBayesClassifier training for sentiment analysis
  10. NLTK Named Entity recognition to a Python list
  11. Computing N Grams using Python
  12. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  13. How do I tokenize a string sentence in NLTK?
  14. How to get rid of punctuation using NLTK tokenizer?
  15. English grammar for parsing in NLTK
  16. NLTK and language detection
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How do I do dependency parsing in NLTK?
  20. training data format for NLTK punkt
  21. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  22. Saving nltk drawn parse tree to image file
  23. Creating a custom categorized corpus in NLTK and Python
  24. tag generation from a text content
  25. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  26. What is NLTK POS tagger asking me to download?
  27. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  28. Fast n-gram calculation
  29. Using the Python NLTK (2.0b5) on the Google App Engine
  30. How to use malt parser in python nltk

Leave a Comment