Faster way to remove stop words in Python

by

Try caching the stopwords object, as shown below. Constructing this each time you call the function seems to be the bottleneck.

    from nltk.corpus import stopwords

    cachedStopWords = stopwords.words("english")

    def testFuncOld():
        text="hello bye the the hi"
        text=" ".join([word for word in text.split() if word not in stopwords.words("english")])

    def testFuncNew():
        text="hello bye the the hi"
        text=" ".join([word for word in text.split() if word not in cachedStopWords])

    if __name__ == "__main__":
        for i in xrange(10000):
            testFuncOld()
            testFuncNew()

I ran this through the profiler: python -m cProfile -s cumulative test.py. The relevant lines are posted below.

nCalls Cumulative Time

10000 7.723 words.py:7(testFuncOld)

10000 0.140 words.py:11(testFuncNew)

So, caching the stopwords instance gives a ~70x speedup.

More Related Contents:

  1. Searching for IP addresses in a file
  2. How to understand regular expression with python?
  3. Escaping regex string
  4. Python regex matching Unicode properties
  5. How can I remove text within parentheses with a regex?
  6. python re.sub group: number after \number
  7. Handling backreferences to capturing groups in re.sub replacement pattern
  8. Python regular expressions – how to capture multiple groups from a wildcard expression?
  9. Named regular expression group “(?Pregexp)”: what does “P” stand for?
  10. Regex matching between two strings?
  11. Capturing group with findall?
  12. Strip / trim all strings of a dataframe
  13. Find out how many times a regex matches in a string in Python
  14. Python re.sub question
  15. How to replace the first occurrence of a regular expression in Python?
  16. Python regular expressions OR
  17. How do I specify a range of unicode characters
  18. Python regex for unicode capitalized words
  19. How to do sed like text replace with python?
  20. RegExp match repeated characters
  21. Replace a string located between
  22. Regex, select closest match
  23. Regex to find words between two tags
  24. How to tell if one regular expression matches a subset of another regular expression?
  25. How to get the first word in the string
  26. Python: UserWarning: This pattern has match groups. To actually get the groups, use str.extract
  27. Split string at commas except when in bracket environment
  28. Using ^ to match beginning of line in Python regex
  29. How to split long regular expression rules to multiple lines in Python
  30. Python regular expression not matching

Leave a Comment