How do I re.search or re.match on a whole file without reading it all into memory?

by

You can use mmap to map the file to memory. The file contents can then be accessed like a normal string:

import re, mmap

with open('/var/log/error.log', 'r+') as f:
  data = mmap.mmap(f.fileno(), 0)
  mo = re.search('error: (.*)', data)
  if mo:
    print "found error", mo.group(1)

This also works for big files, the file content is internally loaded from disk as needed.

More Related Contents:

  1. Speed up millions of regex replacements in Python 3
  2. Mass string replace in python?
  3. Fastest way to write huge data in file
  4. Why does this take so long to match? Is it a bug?
  5. Is explicitly closing files important?
  6. Iterating on a file doesn’t work the second time
  7. Match a whole word in a string using dynamic regex
  8. Why does Python code run faster in a function?
  9. Directory-tree listing in Python
  10. Case insensitive regular expression without re.compile?
  11. Python string.replace regular expression [duplicate]
  12. Regex matching 5-digit substrings not enclosed with digits
  13. check what files are open in Python
  14. Why is looping over range() in Python faster than using a while loop?
  15. Regular expression to extract URL from an HTML link
  16. BeautifulSoup – search by text inside a tag
  17. How to convert a file to utf-8 in Python?
  18. Get the string within brackets in Python
  19. Compare two files report difference in python
  20. Python 2.7: Print to File
  21. why isn’t numpy.mean multithreaded?
  22. How can I make multiple replacements in a string using a dictionary?
  23. Read from File, or STDIN
  24. Is there a way to circumvent Python list.append() becoming progressively slower in a loop as the list grows?
  25. What are the differences between bool() and operator.truth()?
  26. improving python code in Monte Carlo simulation
  27. How to get the first word in the string
  28. differences between “d = dict()” and “d = {}”
  29. Split string at commas except when in bracket environment
  30. numpy faster than numba and cython , how to improve numba code

Leave a Comment