Efficiently count word frequencies in python

by

The most succinct approach is to use the tools Python gives you.

from future_builtins import map  # Only on Python 2

from collections import Counter
from itertools import chain

def countInFile(filename):
    with open(filename) as f:
        return Counter(chain.from_iterable(map(str.split, f)))

That’s it. map(str.split, f) is making a generator that returns lists of words from each line. Wrapping in chain.from_iterable converts that to a single generator that produces a word at a time. Counter takes an input iterable and counts all unique values in it. At the end, you return a dict-like object (a Counter) that stores all unique words and their counts, and during creation, you only store a line of data at a time and the total counts, not the whole file at once.

In theory, on Python 2.7 and 3.1, you might do slightly better looping over the chained results yourself and using a dict or collections.defaultdict(int) to count (because Counter is implemented in Python, which can make it slower in some cases), but letting Counter do the work is simpler and more self-documenting (I mean, the whole goal is counting, so use a Counter). Beyond that, on CPython (the reference interpreter) 3.2 and higher Counter has a C level accelerator for counting iterable inputs that will run faster than anything you could write in pure Python.

Update: You seem to want punctuation stripped and case-insensitivity, so here’s a variant of my earlier code that does that:

from string import punctuation

def countInFile(filename):
    with open(filename) as f:
        linewords = (line.translate(None, punctuation).lower().split() for line in f)
        return Counter(chain.from_iterable(linewords))

Your code runs much more slowly because it’s creating and destroying many small Counter and set objects, rather than .update-ing a single Counter once per line (which, while slightly slower than what I gave in the updated code block, would be at least algorithmically similar in scaling factor).

More Related Contents:

  1. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  2. python kmeans on string
  3. Why does the following code cant get value of n_fold = 1?
  4. How to extract the decision rules from scikit-learn decision-tree?
  5. How to one-hot-encode from a pandas column containing a list?
  6. Ordinal numbers replacement
  7. scikit-learn & statsmodels – which R-squared is correct?
  8. ImportError in importing from sklearn: cannot import name check_build
  9. Error importing scikit-learn modules
  10. Clustering text documents using scikit-learn kmeans in Python
  11. scikit-learn cross validation, negative values with mean squared error
  12. Efficiently create sparse pivot tables in pandas?
  13. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  14. How to get rid of punctuation using NLTK tokenizer?
  15. Passing categorical data to Sklearn Decision Tree
  16. fit-transform on training data and transform on test data [duplicate]
  17. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  18. Getting model attributes from pipeline
  19. Fuzzy String Comparison
  20. How to find the importance of the features for a logistic regression model?
  21. How to use sklearn fit_transform with pandas and return dataframe instead of numpy array?
  22. tag generation from a text content
  23. Got continuous is not supported error in RandomForestRegressor
  24. confused about random_state in decision tree of scikit learn
  25. Implementing Bag-of-Words Naive-Bayes classifier in NLTK
  26. Can I send callbacks to a KerasClassifier?
  27. How to get most informative features for scikit-learn classifier for different class?
  28. scikit-learn .predict() default threshold
  29. ImportError: No module named sklearn.cross_validation
  30. ModuleNotFoundError: No module named ‘sklearn’

Leave a Comment