os.walk very slow, any way to optimise?

by

Yes: use Python 3.5 (which is still currently a RC, but should be out momentarily). In Python 3.5, os.walk was rewritten to be more efficient.

This work done as part of PEP 471.

Extracted from the PEP:

Python’s built-in os.walk() is significantly slower than it needs to
be, because — in addition to calling os.listdir() on each directory
— it executes the stat() system call or GetFileAttributes() on each file to determine whether the entry is a directory or not.

But the underlying system calls — FindFirstFile / FindNextFile on
Windows and readdir on POSIX systems — already tell you whether the
files returned are directories or not, so no further system calls are
needed. Further, the Windows system calls return all the information
for a stat_result object on the directory entry, such as file size and
last modification time.

In short, you can reduce the number of system calls required for a
tree function like os.walk() from approximately 2N to N, where N is
the total number of files and directories in the tree. (And because
directory trees are usually wider than they are deep, it’s often much
better than this.)

In practice, removing all those extra system calls makes os.walk()
about 8-9 times as fast on Windows, and about 2-3 times as fast on
POSIX systems. So we’re not talking about micro-optimizations. See
more benchmarks here.

More Related Contents:

  1. How do I profile a Python script?
  2. Fastest way to list all primes below N
  3. A weighted version of random.choice
  4. How to improve performance of this code?
  5. Using os.walk() to recursively traverse directories in Python
  6. How to do a recursive sub-folder search and return files in a list?
  7. Flatten an irregular list of lists
  8. String concatenation without ‘+’ operator
  9. How to iterate over a list in chunks
  10. When is not a good time to use python generators?
  11. Python equivalent of MATLAB’s “ismember” function
  12. Python vectorizing nested for loops
  13. What does Python optimization (-O or PYTHONOPTIMIZE) do?
  14. Efficient way to take the minimum/maximum n values and indices from a matrix using NumPy
  15. How do I put a constraint on SciPy curve fit?
  16. Processing time gets longer and longer after each iteration (TensorFlow)
  17. Most efficient way to calculate radial profile
  18. scipy is not optimizing and returns “Desired error not necessarily achieved due to precision loss”
  19. Flatten an irregular (arbitrarily nested) list of lists
  20. Filtering os.walk() dirs and files
  21. os.walk without hidden folders
  22. Python: Finding multiple roots of nonlinear equation
  23. Multiplying a huge number times random() (Python)
  24. very quickly getting total size of folder
  25. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  26. Finding the most frequent character in a string
  27. Looking for a fast way to find the polygon a point belongs to using Shapely
  28. Python equivalence to inline functions or macros
  29. Elegant way to remove items from sequence in Python? [duplicate]
  30. Need the path for particular files using os.walk() [duplicate]

Leave a Comment