Extending ImageDataGenerator

[tetmin]

Is there an easy way to write generator extensions for Keras? I'd like to use some of the ImageDataGenerator preprocessing steps but also add some of my own such as randomly occluding areas of the image, adding noise etc. If not I can add some of these things to the ImageDataGenerator class in Keras, is that something that would be useful?

Also can I just double check that the ImageDataGenerator does indeed generate batches of examples in a non-locking way, thereby not causing any GPU training bottlenecks?

[oeway]

I would be interested if someone can make the ImageDataGenerator extendable.

Plus, another feature I would like to have is to separate the random variables generator part which used by ImageDataGenerator, thus the variables can be saved somewhere and make the process reproducible, you don't need to actually store the transformed images, just store the random variables, as a dictionary maybe. These random variables can be feed to into a ImageDataGenerator to produce the results.

This going to be super useful in case of dense prediction(such as segmentation), in which we need to do data augmentation on both input image and output image, we can make two ImageDataGenerator, and use the separated random generator to produce the random variables, and then feed into two ImageDataGenerator, so the spatial transform such as rotation are synchronized on both input and output.

I would begin to work on this, please let me know if anyone else have ideas regarding this.

EDIT: perhaps you would think there is a seed option in flow and flow_from_directory, yes, indeed, you can reproduce the image by controlling the seed, but the problem comes if you want to apply different configuration to two ImageDataGenerator, for example, add noise on the input image but not on the output image, in that case, even you use the same seed, the spatial transformation could be different. In that sense, perhaps, we should use different seed for different type of data augmentation function, what do you think?

[oeway]

I am working on improve ImageDataGenerator but would be great anyone interested could provide suggestions and feedbacks.
The main idea to make it more flexible, and support dense prediction dataset. Here are a bunch of ideas I would like to discuss:

• provide a customizable pipeline( a list of functions) to let user add/change their own preprocessing. There will be a pipeline contains random_transform and standardize by default, and user can insert any function in any position of the pipeline. ( For example, Add target_shape and resize to ImageDataGenerator #2738 can be resolved, and people can use opencv scikit-image or other libraries)
• reproducible random spatial transform by using a separate random seed from the keyword, so dense prediction data has image-in and image-out can use two different ImageDataGenerator which have the same random spatial transform. (related to ImageDataGenerator with masks as labels #3059)
• provide featurewise_standardize_axis and samplewise_standardize_axis for more flexible standardization, for example, you can pass (0, 2, 3) to do a channel-wise normalization(related to issue ImageDataGenerator samplewise_center standardize method subtracting mean over image channels #2559 and PR Update ImageDataGenerator to allow channelwise standardization #2879, also mentioned in Rewriting image augmenter  #2446).

For flow_from_directory():

• remove the limit for channel number and color mode, by using PIL, we can just provide the PIL image mode as a parameter which gives more flexibility on the image format, channel data type and channel number. Image with ARGB mode png file, or F mode(float) tiff which is widely used in many field should be naturally supported. (also related to ImageDataGenerator - feature-request support for 'BGR' #3140)
• provide an optional image file reader keyword to replace the underlying PIL image reader (for flow_from_directory). With a file reader function which takes a path as input and return a numpy array, we can provide much more flexible on the image format. We can read hdf5, npz, pickle file, or for example, you can use a library named openslide to read histopathological images. (also related to ImageDataGenerator for time distributed image data #1819 for reading time distributed data)

For fit():

• fit is required for feature-wise standardization and ZCA , and it only takes an array as parameter, there is no fit for directory. For now, we need to manually read a subset of the image to do this fit for a directory. One idea is we can change fit() to accept the generator itself(flow_from_directory), of course, standardization should be disabled during fit.

[pengpaiSH]

@oeway When it comes to visual semantic segmentation, I find out that extending ImageDataGenerator is really necessary! Current ImageDataGenerator will only conduct transformations on X instead of Y. However, in image segmentation scenarios, input image X and output mask Y should be transformed simultaneously.

[oeway]

@pengpaiSH I have a working version right now, you are welcome to try it out. https://github.com/oeway/keras/tree/extendImageDataGenerator (branch: extendImageDataGenerator)

For now, you can customize the pipeline and synchronize two ImageDataGenerator by a __add__(+) operator. You can see more extensions in the following example:

from keras.preprocessing.image import ImageDataGenerator,standardize,random_transform
# input generator with standardization on
datagenX = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    featurewise_standardize_axis=(0, 2, 3),
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0,
    verbose=1)

# output generator with standardization off
datagenY = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0)

def center_crop(x, center_crop_size, **kwargs):
    centerw, centerh = x.shape[1]//2, x.shape[2]//2
    halfw, halfh = center_crop_size[0]//2, center_crop_size[1]//2
    return x[:, centerw-halfw:centerw+halfw,centerh-halfh:centerh+halfh]

def random_crop(x, random_crop_size, sync_seed=None, **kwargs):
    np.random.seed(sync_seed)
    w, h = x.shape[1], x.shape[2]
    rangew = (w - random_crop_size[0]) // 2
    rangeh = (h - random_crop_size[1]) // 2
    offsetw = 0 if rangew == 0 else np.random.randint(rangew)
    offseth = 0 if rangeh == 0 else np.random.randint(rangeh)
    return x[:, offsetw:offsetw+random_crop_size[0], offseth:offseth+random_crop_size[1]]

datagenX.config['random_crop_size'] = (800, 800)
datagenY.config['random_crop_size'] = (800, 800)
datagenX.config['center_crop_size'] = (512, 512)
datagenY.config['center_crop_size'] = (360, 360)

# customize the pipeline
datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop])
datagenY.set_pipeline([random_crop, random_transform, center_crop])

# flow from directory is extended to support more format and also you can even use your own reader function
# here is an example of reading image data saved in csv file
# datagenX.flow_from_directory(csvFolder, image_reader=csvReaderGenerator, read_formats={'csv'}, reader_config={'target_size':(572,572),'resolution':20, 'crange':(0,100)}, class_mode=None, batch_size=1)

dgdx= datagenX.flow_from_directory(inputDir, class_mode=None, read_formats={'png'}, batch_size=2)
dgdy= datagenY.flow_from_directory(outputDir,  class_mode=None, read_formats={'png'}, batch_size=2)

# you can now fit a generator as well
datagenX.fit_generator(dgdx, nb_iter=100)

# here we sychronize two generator and combine it into one
train_generator = dgdx+dgdy

model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)

[pengpaiSH]

@oeway Thank you for your quick response and your enhancement for ImageGenerator which I think is really really necessary. Below are my confusions:

1. As far as I understand, center_crop and random_crop is our customized augmented transformation, right?
2. In the defined pipeline, datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop]), where does random_transform and standardize come from? They are not even defined such as center_crop.
3. I don't understand why datagenX should be fit_generator()? I think only model should have fit_generator()

[oeway]

Sorry for the confusion, I will write documents about it when I have
time.

1, yes, you just need to provide a function and put in the pipeline,
center_crop is one example, it shows how you can define the function and
how to set arguments for your function.

2, I forgot to put the import line, so random_transform and standardize are
to predefined function which I split from the previous ImageDataGenerator,
they are from the same module as ImageDataGenerator. Just do:

from keras.preprocessing.image import standardize,random_transform

3, fit_generator and fit is for some augment function such as feature_wise
standardization which require a pre computed mean and std value. I added
fit_generator for flow_from_directory(see my previous post for why we need
that), and I generalized the fit interface so that you can now add
customized function which support pre-fit.

I will document all the features if there are more people interested, and
perhaps make a PR. In the meantime, welcome to try and provide feedback.

On Tue, Aug 9, 2016 at 10:49 Pai Peng notifications@github.com wrote:

@oeway https://github.com/oeway Thank you for your quick response and
your enhancement for ImageGenerator which I think is really really
necessary. Below are my confusions:

1. As far as I understand, center_crop and random_crop is our
   customized augmented transformation, right?
2. In the defined pipeline, datagenX.set_pipeline([random_crop,
   random_transform, standardize, center_crop]), where does
   random_transform and standardize come from? They are not even defined
   such as center_crop.
3. I don't understand why datagenX should be fit_generator()? I think
   only model should have fit_generator()

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3338 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdNy2ZFmGvxVU_i2jfVxSHOREIHs7dGks5qeD8ogaJpZM4JXAo4
.

[pengpaiSH]

@oeway Thank you for your clarifying. Perfect! By the way, the transformations defined in the pipeline come in order or not ?

[oeway]

@pengpaiSH Welcome, of course, the idea is use an ordered list to defined the pipeline, and you can change the order of the pipeline.
Let me know if you have any problem.

[landersson]

I also need random cropping, better scaling etc and was just about to start implementing something like this myself. Looks very useful. Will you try to get this merged into the main repo?

[burgalon]

Any plans to merge @oeway branch? I'm specifically interested in the CSV reader option

[ncullen93]

@burgalon It's very straight-forward .. like maybe 10 lines of code.. to extend the data generator to read csv.. Still agree it should be merged but in case you dont want to wait :) . I attached some very unpolished code which works perfectly for me (see 'flow_np_from_directory' function and 'NumpyDirectoryIterator' class specifically) to read in .npy files in the same directory structure as the image version, e.g.:

• dir/
  -class1/
  ... file1.npy
  ... file2.npy
  -class2/
  ... file3.npy

image.zip

[burgalon]

Thanks @ncullen93