a python library that accepts some text, and replaces phone numbers, names, and so on with tokens

by

As Harrison pointed out, nltk has named entity recognition, which is what you want for this task. Here is a good sample to get you started.

From the site:

import nltk 

sentences = nltk.sent_tokenize(text)
tokenized_sentences =      [nltk.word_tokenize(sentence) for sentence in sentences]
tagged_sentences = [nltk.pos_tag(sentence) for sentence in tokenized_sentences]
chunked_sentences = nltk.ne_chunk_sents(tagged_sentences, binary=True)

def extract_entity_names(t):
    entity_names = []

    if hasattr(t, 'label') and t.label:
        if t.label() == 'NE':
            entity_names.append(' '.join([child[0] for child in t]))
        else:
            for child in t:
                entity_names.extend(extract_entity_names(child))

    return entity_names

entity_names = []
for tree in chunked_sentences:
    # Print results per sentence
    # print extract_entity_names(tree)

    entity_names.extend(extract_entity_names(tree))

# Print all entity names
#print entity_names

# Print unique entity names
print set(entity_names)

More Related Contents:

  1. Can you make that thats code run in python 2.7 from 3.5 [closed]
  2. Python sort multi dimensional dict
  3. How to write to an existing excel file without overwriting data (using pandas)?
  4. How to set time limit on raw_input
  5. Cannot find module cv2 when using OpenCV
  6. Unable to locate element of credit card number using selenium python
  7. Unnamed Python objects have the same id
  8. Move an object every few seconds in Pygame
  9. Intersection of two lists including duplicates?
  10. bs4.FeatureNotFound: Couldn’t find a tree builder with the features you requested: lxml. Do you need to install a parser library?
  11. How can I convert 24 hour time to 12 hour time?
  12. Defining “boolness” of a class in python
  13. AttributeError: ‘datetime’ module has no attribute ‘strptime’
  14. When calling super() in a derived class, can I pass in self.__class__? [duplicate]
  15. Import psycopg2 Library not loaded: libssl.1.0.0.dylib
  16. Clustering text documents using scikit-learn kmeans in Python
  17. How to pass dictionary as command line argument to Python script?
  18. ImportError on python 3, worked fine on python 2.7
  19. combine word document using python docx
  20. How do I flatten a list of lists/nested lists? [duplicate]
  21. ImportError: No module named bz2 for Python 2.7.2
  22. Converting .jpg images to .png
  23. Why do list comprehensions write to the loop variable, but generators don’t? [duplicate]
  24. TypeError: super() takes at least 1 argument (0 given) error is specific to any python version?
  25. If list index exists, do X
  26. Python list greater than number [duplicate]
  27. Install Beautiful Soup using pip [duplicate]
  28. Why does “%-d”, or “%-e” remove the leading space or zero?
  29. Decorator error: NoneType object is not callable
  30. How to measure download speed and progress using requests?

Leave a Comment