Use a generator for Keras model.fit_generator

by

I can’t help debug your code since you didn’t post it, but I abbreviated a custom data generator I wrote for a semantic segmentation project for you to use as a template:

def generate_data(directory, batch_size):
    """Replaces Keras' native ImageDataGenerator."""
    i = 0
    file_list = os.listdir(directory)
    while True:
        image_batch = []
        for b in range(batch_size):
            if i == len(file_list):
                i = 0
                random.shuffle(file_list)
            sample = file_list[i]
            i += 1
            image = cv2.resize(cv2.imread(sample[0]), INPUT_SHAPE)
            image_batch.append((image.astype(float) - 128) / 128)

        yield np.array(image_batch)

Usage:

model.fit_generator(
    generate_data('~/my_data', batch_size),
    steps_per_epoch=len(os.listdir('~/my_data')) // batch_size)

More Related Contents:

  1. How to add attention layer to a Bi-LSTM
  2. If range() is a generator in Python 3.3, why can I not call next() on a range?
  3. Tensorflow (.pb) format to Keras (.h5)
  4. How do I create a variable-length input LSTM in Keras?
  5. ImportError: libcublas.so.9.0: cannot open shared object file
  6. Keras, Tensorflow : Merge two different model output into one
  7. Can’t save custom subclassed model
  8. How to Convert python list into nested dictionaries in pythonic way [closed]
  9. zip variable empty after first use
  10. How to print a generator expression?
  11. Cleanest way to get last item from Python iterator
  12. No multiprocessing print outputs (Spyder)
  13. Clear PyCharm Run Window
  14. Wait page to load before getting data with requests.get in python 3
  15. python 3.0 open() default encoding
  16. How to use Tor with Chrome browser through Selenium
  17. Selenium “Unable to find a matching set of capabilities” despite driver being in /usr/local/bin
  18. Why does next raise a ‘StopIteration’, but ‘for’ do a normal return?
  19. ‘python3’ is not recognized as an internal or external command, operable program or batch file
  20. Failed to load resource: the server responded with a status of 429 (Too Many Requests) and 404 (Not Found) with ChromeDriver Chrome through Selenium
  21. ModuleNotFoundError: No module named ‘__main__.xxxx’; ‘__main__’ is not a package
  22. Convert object data type to string issue in python
  23. Python ModuleNotFoundError while importing a module in conftest for Pytest framework
  24. Pandas pd.cut() – binning datetime column / series
  25. How to install pip for Python 3.6 on Ubuntu 16.10?
  26. sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table
  27. Google Cloud Functions Python Logging issue
  28. itertools.cycle().next()?
  29. WinError 10061 – No Connection Could be made
  30. using Subprocess to avoid long-running task from disconnecting discord.py bot?

Leave a Comment