Memory error on using data generator in keras

Question

I am using the following augmentations on dataset of size 9 GB:

datagen = ImageDataGenerator(
    featurewise_center=False,  # set input mean to 0 over the dataset
    samplewise_center=False,  # set each sample mean to 0
    featurewise_std_normalization=False,  # divide inputs by std of the dataset
    samplewise_std_normalization=False,  # divide each input by its std
    zca_whitening=True,  # apply ZCA whitening
    rotation_range=30,  # randomly rotate images in the range (degrees, 0 to 180)
    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
    horizontal_flip=True,  # randomly flip images
    vertical_flip=True)  # randomly flip images
datagen.fit(a)

model.fit_generator(datagen.flow(a,b, batch_size=32),
                    steps_per_epoch=len(a) / 32, epochs=epochs, class_weight = sclass_weight, validation_data = [c, d],callbacks = [MetricsCheckpoint('logs')])

When the code comes to the datagen.fit, I get into memory error ( the code doesn't even go into training)

I have 50 gb ram and am training it on a K80 with a batch size of 32, so don't think that will be a problem.

It works fine when I comment all the augmentations.

Can someone tell me where I am going wrong?

You are running out of memory, because the computation are done on the GPU have a look at your house memory — Aditya, May 31 '18 at 05:13
As I said earlier, gpu has 12 GB memory, and a batch size of 32 shouldn't be a problem right — Srihari, May 31 '18 at 05:24
https://stackoverflow.com/q/49458905/6524169 this might help — Aditya, May 31 '18 at 06:08
@Media No, we got it here at workplace. We use a scheduler so we can request 50 gigs memory. Hence the k80 as slaves. Hope its clear now — Srihari, May 31 '18 at 07:53

score 1 · Answer 1 · answered Aug 11 '19 at 17:01

It seems like the root cause for this is that zca_whitening is set to True. According to the answer from the keras-collaborator rragundez in this related Github issue, there is no work-around for it, except disabling ZCA whitening:

This is known problem with the ZCA method. There is no real immediate way around it. The problem is with the calculation of the sigma matrix which is a dot product.

The other alternative to disabling it would be to use images with a smaller resolution. But I suppose that's rarely a practical solution.

Memory error on using data generator in keras

1 Answers1