Checking for corrupted files in directory with hundreds of thousands of images gradually slows down

by

If you have that many images, I would suggest you use multiprocessing. I created 100,000 files of which 5% were corrupt and checked them like this:

#!/usr/bin/env python3

import glob
from multiprocessing import Pool
from PIL import Image

def CheckOne(f):
    try:
        im = Image.open(f)
        im.verify()
        im.close()
        # DEBUG: print(f"OK: {f}")
        return
    except (IOError, OSError, Image.DecompressionBombError):
        # DEBUG: print(f"Fail: {f}")
        return f

if __name__ == '__main__':
    # Create a pool of processes to check files
    p = Pool()

    # Create a list of files to process
    files = [f for f in glob.glob("*.jpg")]

    print(f"Files to be checked: {len(files)}")

    # Map the list of files to check onto the Pool
    result = p.map(CheckOne, files)

    # Filter out None values representing files that are ok, leaving just corrupt ones
    result = list(filter(None, result)) 
    print(f"Num corrupt files: {len(result)}")

Sample Output

Files to be checked: 100002
Num corrupt files: 5001

That takes 1.6 seconds on my 12-core CPU with NVME disk, but should still be noticeably faster for you.

More Related Contents:

  1. How do I resize an image using PIL and maintain its aspect ratio?
  2. Combine several images horizontally with Python
  3. How to reduce a jpeg size to a ‘desired size’?
  4. How to install PIL with pip on Mac OS?
  5. How to write PNG image to string with the PIL?
  6. Trim whitespace using PIL
  7. How do I generate circular thumbnails with PIL?
  8. Image.open() cannot identify image file – Python?
  9. How to use PIL to make all white pixels transparent?
  10. PIL: Convert Bytearray to Image
  11. ImportError: cannot import name _imaging
  12. PIL open() method not working with BytesIO
  13. Convert png to jpeg using Pillow
  14. How to send image generated by PIL to browser?
  15. How to convert PIL Image.image object to base64 string? [duplicate]
  16. Package (Python PIL/Pillow) installed but I can’t import it
  17. How do I read image data from a URL?
  18. Load just part of an image in python
  19. python tkinter how to bind key to a button
  20. Converting a PDF to a series of images with Python
  21. Detecting thresholds in HSV color space (from RGB) using Python / PIL
  22. Python imaging library show() on Windows
  23. How to add all images from a folder to a button (in my frame) [closed]
  24. PIL “IOError: image file truncated” with big images
  25. How do I create an image in PIL using a list of RGB tuples?
  26. How to CREATE a transparent gif (or png) with PIL (python-imaging)
  27. Automatically cropping an image with python/PIL
  28. Python Image Library: How to combine 4 images into a 2 x 2 grid?
  29. Resize image in Python without losing EXIF data
  30. How do I draw text at an angle using python’s PIL?

Leave a Comment