Difference between Python’s collections.Counter and nltk.probability.FreqDist

by

nltk.probability.FreqDist is a subclass of collections.Counter.

From the docs:

A frequency distribution for the outcomes of an experiment. A
frequency distribution records the number of times each outcome of an
experiment has occurred. For example, a frequency distribution could
be used to record the frequency of each word type in a document.
Formally, a frequency distribution can be defined as a function
mapping from each sample to the number of times that sample occurred
as an outcome.

The inheritance is explicitly shown from the code and essentially, there’s no difference in terms of how a Counter and FreqDist is initialized, see https://github.com/nltk/nltk/blob/develop/nltk/probability.py#L106

So speed-wise, creating a Counter and FreqDist should be the same. The difference in speed should be insignificant but it’s good to note that the overheads could be:

  • the compilation of the class in when defining it in an interpreter
  • the cost of duck-typing .__init__()

The major difference is the various functions that FreqDist provides for statistical / probabilistic Natural Language Processing (NLP), e.g. finding hapaxes. The full list of functions that FreqDist extends Counter are as followed:

>>> from collections import Counter
>>> from nltk import FreqDist
>>> x = FreqDist()
>>> y = Counter()
>>> set(dir(x)).difference(set(dir(y)))
set(['plot', 'hapaxes', '_cumulative_frequencies', 'r_Nr', 'pprint', 'N', 'unicode_repr', 'B', 'tabulate', 'pformat', 'max', 'Nr', 'freq', '__unicode__'])

When it comes to using FreqDist.most_common(), it’s actually using the parent function from Counter so the speed of retrieving the sorted most_common list is the same for both types.

Personally, when I just want to retrieve counts, I use collections.Counter. But when I need to do some statistical manipulation, I either use nltk.FreqDist or I would dump the Counter into a pandas.DataFrame (see Transform a Counter object into a Pandas DataFrame).

More Related Contents:

  1. How to use Stanford Parser in NLTK using Python
  2. Creating a new corpus with NLTK
  3. Python NLTK pos_tag not returning the correct part-of-speech tag
  4. Stopword removal with NLTK
  5. Ordinal numbers replacement
  6. How to config nltk data directory from code?
  7. Classification using movie review corpus in NLTK/Python
  8. Convert words between verb/noun/adjective forms
  9. nltk NaiveBayesClassifier training for sentiment analysis
  10. NLTK Named Entity recognition to a Python list
  11. Computing N Grams using Python
  12. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  13. How do I tokenize a string sentence in NLTK?
  14. How to get rid of punctuation using NLTK tokenizer?
  15. English grammar for parsing in NLTK
  16. NLTK and language detection
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How do I do dependency parsing in NLTK?
  20. training data format for NLTK punkt
  21. Saving nltk drawn parse tree to image file
  22. Creating a custom categorized corpus in NLTK and Python
  23. tag generation from a text content
  24. NLTK WordNet Lemmatizer: Shouldn’t it lemmatize all inflections of a word?
  25. What is NLTK POS tagger asking me to download?
  26. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  27. Fast n-gram calculation
  28. Using the Python NLTK (2.0b5) on the Google App Engine
  29. How to use malt parser in python nltk
  30. How to tweak the NLTK sentence tokenizer

Leave a Comment