How to read tokens without reading whole line or file

by

To read tokens from a file one by one; you could use re module to generate tokens from a memory-mapped file:

#!/usr/bin/env python3
import re
import sys
from mmap import ACCESS_READ, mmap    

def generate_tokens(filename, pattern):
    with open(filename) as f, mmap(f.fileno(), 0, access=ACCESS_READ) as mm:
         yield from re.finditer(pattern, mm)

# sum all integers in a file specified at the command-line
print(sum(int(m.group()) for m in generate_tokens(sys.argv[1], br'\d+')))

It works even if the file doesn’t fit in memory.

More Related Contents:

  1. How do I delete a file or folder in Python?
  2. Lazy Method for Reading Big File in Python?
  3. Reading binary file and looping over each byte
  4. How do I concatenate text files in Python?
  5. Is it possible to modify lines in a file in-place?
  6. open() gives FileNotFoundError/IOError: Errno 2 No such file or directory
  7. How to open a file using the open with statement
  8. Reading from a frequently updated file
  9. How to write a multidimensional array to a text file?
  10. Read file from line 2 or skip header row
  11. Python 32-bit memory limits on 64bit windows
  12. Python read JSON file and modify
  13. Writing a list to a file with Python, with newlines
  14. Read from a log file as it’s being written using python
  15. File read using “open()” vs “with open()” [duplicate]
  16. Create file but if name exists add number
  17. How to open a password protected excel file using python?
  18. pandas to_csv output quoting issue
  19. Python Pandas: How to read only first n rows of CSV files in?
  20. How to safely open/close files in python 2.4
  21. python close file descriptor question
  22. ValueError : I/O operation on closed file
  23. Parsing apache log files
  24. How can I read a single character at a time from a file in Python?
  25. Trying to use open(filename, ‘w’ ) gives IOError: [Errno 2] No such file or directory if directory doesn’t exist
  26. Behaviour of truncate() method in Python
  27. How to read lines from a file in python starting from the end
  28. Convert percent string to float in pandas read_csv
  29. Speed up writing to files
  30. Change file to read-only mode in Python

Leave a Comment