a python library that accepts some text, and replaces phone numbers, names, and so on with tokens

As Harrison pointed out, nltk has named entity recognition, which is what you want for this task. Here is a good sample to get you started.

From the site:

import nltk 

sentences = nltk.sent_tokenize(text)
tokenized_sentences =      [nltk.word_tokenize(sentence) for sentence in sentences]
tagged_sentences = [nltk.pos_tag(sentence) for sentence in tokenized_sentences]
chunked_sentences = nltk.ne_chunk_sents(tagged_sentences, binary=True)

def extract_entity_names(t):
    entity_names = []

    if hasattr(t, 'label') and t.label:
        if t.label() == 'NE':
            entity_names.append(' '.join([child[0] for child in t]))
            for child in t:

    return entity_names

entity_names = []
for tree in chunked_sentences:
    # Print results per sentence
    # print extract_entity_names(tree)


# Print all entity names
#print entity_names

# Print unique entity names
print set(entity_names)

Browse More Popular Posts

Leave a Comment