very quickly getting total size of folder

by

You are at a disadvantage.

Windows Explorer almost certainly uses FindFirstFile/FindNextFile to both traverse the directory structure and collect size information (through lpFindFileData) in one pass, making what is essentially a single system call per file.

Python is unfortunately not your friend in this case. Thus,

  1. os.walk first calls os.listdir (which internally calls FindFirstFile/FindNextFile)
    • any additional system calls made from this point onward can only make you slower than Windows Explorer
  2. os.walk then calls isdir for each file returned by os.listdir (which internally calls GetFileAttributesEx — or, prior to Win2k, a GetFileAttributes+FindFirstFile combo) to redetermine whether to recurse or not
  3. os.walk and os.listdir will perform additional memory allocation, string and array operations etc. to fill out their return value
  4. you then call getsize for each file returned by os.walk (which again calls GetFileAttributesEx)

That is 3x more system calls per file than Windows Explorer, plus memory allocation and manipulation overhead.

You can either use Anurag’s solution, or try to call FindFirstFile/FindNextFile directly and recursively (which should be comparable to the performance of a cygwin or other win32 port du -s some_directory.)

Refer to os.py for the implementation of os.walk, posixmodule.c for the implementation of listdir and win32_stat (invoked by both isdir and getsize.)

Note that Python’s os.walk is suboptimal on all platforms (Windows and *nices), up to and including Python3.1. On both Windows and *nices os.walk could achieve traversal in a single pass without calling isdir since both FindFirst/FindNext (Windows) and opendir/readdir (*nix) already return file type via lpFindFileData->dwFileAttributes (Windows) and dirent::d_type (*nix).

Perhaps counterintuitively, on most modern configurations (e.g. Win7 and NTFS, and even some SMB implementations) GetFileAttributesEx is twice as slow as FindFirstFile of a single file (possibly even slower than iterating over a directory with FindNextFile.)

Update: Python 3.5 includes the new PEP 471 os.scandir() function that solves this problem by returning file attributes along with the filename. This new function is used to speed up the built-in os.walk() (on both Windows and Linux). You can use the scandir module on PyPI to get this behavior for older Python versions, including 2.x.

More Related Contents:

  1. How can I safely create a nested directory?
  2. How do I delete a file or folder in Python?
  3. A weighted version of random.choice
  4. Importing modules from parent folder
  5. How can I iterate over files in a given directory?
  6. How do you get a directory listing sorted by creation date in python?
  7. Directory-tree listing in Python
  8. What is the fastest way to flatten arbitrarily nested lists in Python? [duplicate]
  9. Iterating through directories with Python
  10. How to find the real user home directory using python?
  11. How to split/partition a dataset into training and test datasets for, e.g., cross validation?
  12. Python equivalent of MATLAB’s “ismember” function
  13. Reading the target of a .lnk file in Python?
  14. Python vectorizing nested for loops
  15. How to efficiently use MySQLDB SScursor?
  16. What does Python optimization (-O or PYTHONOPTIMIZE) do?
  17. How do I put a constraint on SciPy curve fit?
  18. Processing time gets longer and longer after each iteration (TensorFlow)
  19. How to select a directory and store the location using tkinter in Python
  20. Import from sibling directory
  21. Most efficient way to calculate radial profile
  22. Deleting read-only directory in Python
  23. Flatten an irregular (arbitrarily nested) list of lists
  24. Fastest Way to Delete a Line from Large File in Python
  25. Multiplying a huge number times random() (Python)
  26. How can I extract the folder path from file path in Python?
  27. Python equivalence to inline functions or macros
  28. Creating files and directories via Python
  29. scipy.optimize.leastsq with bound constraints
  30. Handling directories with spaces Python subprocess.call()

Leave a Comment