training data format for NLTK punkt

by

Ah yes, Punkt tokenizer is the magical unsupervised sentence boundary detection. And the author’s last name is pretty cool too, Kiss and Strunk (2006). The idea is to use NO annotation to train a sentence boundary detector, hence the input will be ANY sort of plaintext (as long as the encoding is consistent).

To train a new model, simply use:

import nltk.tokenize.punkt
import pickle
import codecs
tokenizer = nltk.tokenize.punkt.PunktSentenceTokenizer()
text = codecs.open("someplain.txt","r","utf8").read()
tokenizer.train(text)
out = open("someplain.pk","wb")
pickle.dump(tokenizer, out)
out.close()

To achieve higher precision and allow you to stop training at any time and still save a proper pickle for your tokenizer, do look at this code snippet for training a German sentence tokenizer, https://github.com/alvations/DLTK/blob/master/dltk/tokenize/tokenizer.py :

def train_punktsent(trainfile, modelfile):
  """ Trains an unsupervised NLTK punkt sentence tokenizer. """
  punkt = PunktTrainer()
  try:
    with codecs.open(trainfile, 'r','utf8') as fin:
      punkt.train(fin.read(), finalize=False, verbose=False)
  except KeyboardInterrupt:
    print 'KeyboardInterrupt: Stopping the reading of the dump early!'
  ##HACK: Adds abbreviations from rb_tokenizer.
  abbrv_sent = " ".join([i.strip() for i in \
                         codecs.open('abbrev.lex','r','utf8').readlines()])
  abbrv_sent = "Start"+abbrv_sent+"End."
  punkt.train(abbrv_sent,finalize=False, verbose=False)
  # Finalize and outputs trained model.
  punkt.finalize_training(verbose=True)
  model = PunktSentenceTokenizer(punkt.get_params())
  with open(modelfile, mode="wb") as fout:
    pickle.dump(model, fout, protocol=pickle.HIGHEST_PROTOCOL)
  return model

However do note that the period detection is very sensitive to the latin fullstop, question mark and exclamation mark. If you’re going to train a punkt tokenizer for other languages that doesn’t use latin orthography, you’ll need to somehow hack the code to use the appropriate sentence boundary punctuation. If you’re using NLTK’s implementation of punkt, edit the sent_end_chars variable.

There are pre-trained models available other than the ‘default’ English tokenizer using nltk.tokenize.sent_tokenize(). Here they are: https://github.com/evandrix/nltk_data/tree/master/tokenizers/punkt

Edited

Note the pre-trained models are currently not available because the nltk_data github repo listed above has been removed.

More Related Contents:

  1. How to use Stanford Parser in NLTK using Python
  2. Creating a new corpus with NLTK
  3. Python NLTK pos_tag not returning the correct part-of-speech tag
  4. Stopword removal with NLTK
  5. Ordinal numbers replacement
  6. How to config nltk data directory from code?
  7. Classification using movie review corpus in NLTK/Python
  8. Convert words between verb/noun/adjective forms
  9. nltk NaiveBayesClassifier training for sentiment analysis
  10. NLTK Named Entity recognition to a Python list
  11. Computing N Grams using Python
  12. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  13. How do I tokenize a string sentence in NLTK?
  14. How to get rid of punctuation using NLTK tokenizer?
  15. English grammar for parsing in NLTK
  16. NLTK and language detection
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How do I do dependency parsing in NLTK?
  20. Difference between Python’s collections.Counter and nltk.probability.FreqDist
  21. Saving nltk drawn parse tree to image file
  22. Creating a custom categorized corpus in NLTK and Python
  23. tag generation from a text content
  24. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  25. What is NLTK POS tagger asking me to download?
  26. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  27. Fast n-gram calculation
  28. Using the Python NLTK (2.0b5) on the Google App Engine
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment