Extract list of Persons and Organizations using Stanford NER Tagger in NLTK

by

Thanks to the link discovered by @Vaulstein, it is clear that the trained Stanford tagger, as distributed (at least in 2012) does not chunk named entities. From the accepted answer:

Many NER systems use more complex labels such as IOB labels, where codes like B-PERS indicates where a person entity starts. The CRFClassifier class and feature factories support such labels, but they’re not used in the models we currently distribute (as of 2012)

You have the following options:

  1. Collect runs of identically tagged words; e.g., all adjacent words tagged PERSON should be taken together as one named entity. That’s very easy, but of course it will sometimes combine different named entities. (E.g. New York, Boston [and] Baltimore is about three cities, not one.) Edit: This is what Alvas’s code does in the accepted anwser. See below for a simpler implementation.

  2. Use nltk.ne_chunk(). It doesn’t use the Stanford recognizer but it does chunk entities. (It’s a wrapper around an IOB named entity tagger).

  3. Figure out a way to do your own chunking on top of the results that the Stanford tagger returns.

  4. Train your own IOB named entity chunker (using the Stanford tools, or the NLTK’s framework) for the domain you are interested in. If you have the time and resources to do this right, it will probably give you the best results.

Edit: If all you want is to pull out runs of continuous named entities (option 1 above), you should use itertools.groupby:

from itertools import groupby
for tag, chunk in groupby(netagged_words, lambda x:x[1]):
    if tag != "O":
        print("%-12s"%tag, " ".join(w for w, t in chunk))

If netagged_words is the list of (word, type) tuples in your question, this produces:

PERSON       Rami Eid
ORGANIZATION Stony Brook University
LOCATION     NY

Note again that if two named entities of the same type occur right next to each other, this approach will combine them. E.g. New York, Boston [and] Baltimore is about three cities, not one.

More Related Contents:

  1. How to use Stanford Parser in NLTK using Python
  2. NLTK Named Entity recognition to a Python list
  3. How to add a label to all words in a file? [closed]
  4. How to remove English words from a file containing Dari words?
  5. How to check if a word is an English word with Python?
  6. How to config nltk data directory from code?
  7. Classification using movie review corpus in NLTK/Python
  8. n-grams in python, four, five, six grams?
  9. Convert words between verb/noun/adjective forms
  10. nltk NaiveBayesClassifier training for sentiment analysis
  11. re.sub erroring with “Expected string or bytes-like object”
  12. All synonyms for word in python? [duplicate]
  13. NLTK-based text processing with pandas
  14. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  15. nltk doesn’t add $NLTK_DATA to search path?
  16. str.translate gives TypeError – Translate takes one argument (2 given), worked in Python 2
  17. Fast/Optimize N-gram implementations in python
  18. Stanford nlp for python
  19. Counting the Frequency of words in a pandas data frame
  20. How to get all the hyponyms of a word/synset in python nltk and wordnet?
  21. Spell Checker for Python
  22. Tokenize a paragraph into sentence and then into words in NLTK
  23. tag generation from a text content
  24. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  25. Fast n-gram calculation
  26. Using the Python NLTK (2.0b5) on the Google App Engine
  27. How to use malt parser in python nltk
  28. Getting 405 error while trying to download nltk data
  29. How to get most informative features for scikit-learn classifier for different class?
  30. NLTK download SSL: Certificate verify failed

Leave a Comment