Tag Archives: ImageDataGenerator

Multi-Label Classification

In the previous blogs, we discussed binary and multi-class classification problems. Both of these are almost similar. The basic assumption underlying these two problems is that each image can contain only one class. For instance, for the dogs vs cats classification, it was assumed that the image can contain either cat or dog but not both. So, in this blog, we will discuss the case where more than one classes can be present in a single image. This type of classification is known as Multi-label classification. Below picture explains this concept beautifully.

Some of the most common techniques for solving multi-label classification problems are

Problem Transformation
Adapted Algorithm
Ensemble approaches

Here, we will only discuss only Binary Relevance, a method that falls under the Problem Transformation category. If you are curious about other methods, you can read this amazing review paper.

In binary relevance, we try to break the problem into a number of binary classification problems. So, now for each class available, we will ask if it is present in the image or not. As we already know that the binary classification uses ‘sigmoid‘ as the last layer activation function and ‘binary_crossentropy‘ as the loss function. So, here we will also use the same. Rest all things are the same.

Now, let’s take a dataset and see how to implement multi-label classification.

Problem Definition

Here, we will take the most common Movie Genre classification based on the poster images problem. Because a movie can belong to more than one genre, for instance, comedy, romance, etc. and hence is a multi-label classification problem.

Dataset

You can download the original dataset from here. This contains two files.

Movie_Poster_Dataset.zip – The poster images
Movie_Poster_Metadata.zip – Metadata of each poster image like ID, genres, box office, etc.

To prepare the dataset, we need images and corresponding genre information. For this, we need to extract the genre information from the Movie_Poster_Metadata.zip file corresponding to each poster image. Let’s see how to do this.

Note: This dataset contains some missing items. For instance, check the “1982” folder in the Movie_Poster_Dataset.zip and Movie_Poster_Metadata.zip. The number of poster images and the corresponding genre information is missing for some movies. So, we need to perform EDA and remove these files.

Steps to perform EDA:

First, we will extract the movie name and corresponding genre information from the Movie_Poster_Metadata.zip file and create a Pandas dataframe using these.
Then we will loop over the poster images in the Movie_Poster_Dataset.zip file and check if it is present in the dataframe created above. If the poster is not present, we will remove that movie from the dataframe.

These two steps will ensure that we are only left with movies that have poster images and genre information. Below is the code for this.

Because the encoding of some files is different, that’s why 2 for loops. Below are the steps performed in the code.

First, open the metadata file
Read line by line
Extract the information corresponding to the ‘Genre’ and ‘imdbID’
Append them into the list and create a dataframe

b = []
b2 = []
for i in range(1980,1982):
    with open('D:/downloads/Movie_Poster_Dataset/groundtruth/{}.txt'.format(i), mode="r") as f:
        for lines in f.readlines():
            lines = lines.rstrip('\n')
            if 'imdbID' in lines:
                a2,b3,c2 = lines.partition(':')
                c2 = c2.lstrip('  "')
                c2 = c2.rstrip('",\n')
                b2.append(c2+'.jpg')

            if "Genre" in lines:
#                 print(lines)
                a,b1,c = lines.partition(':')
                c = c.lstrip('  "')
                c = c.rstrip('",\n')
                c1 = c.split(',')
                c1 = map(str.strip, c1) 
                b.append(list(c1))
        f.close()
        
for i in range(1982,2016):
    with open('D:/downloads/Movie_Poster_Dataset/groundtruth/{}.txt'.format(i), mode="r",encoding='utf-16-le') as f:
        for lines in f.readlines():
            lines = lines.rstrip('\n')
            if 'imdbID' in lines:
                a2,b3,c2 = lines.partition(':')
                c2 = c2.lstrip('  "')
                c2 = c2.rstrip('",\n')
                b2.append(c2+'.jpg')                
            if "Genre" in lines:
                a,b1,c = lines.partition(':')
                c = c.lstrip('  "')
                c = c.rstrip('",\n')
                c1 = c.split(',')
                c1 = map(str.strip, c1) 
                b.append(list(c1))
        f.close()

data = pd.DataFrame({'name':b2,'filename':b})

b = []

b2 = []

for i in range(1980,1982):

with open('D:/downloads/Movie_Poster_Dataset/groundtruth/{}.txt'.format(i), mode="r") as f:

for lines in f.readlines():

lines = lines.rstrip('\n')

if 'imdbID' in lines:

a2,b3,c2 = lines.partition(':')

c2 = c2.lstrip(' "')

c2 = c2.rstrip('",\n')

b2.append(c2+'.jpg')

if "Genre" in lines:

# print(lines)

a,b1,c = lines.partition(':')

c = c.lstrip(' "')

c = c.rstrip('",\n')

c1 = c.split(',')

c1 = map(str.strip, c1)

b.append(list(c1))

f.close()

for i in range(1982,2016):

with open('D:/downloads/Movie_Poster_Dataset/groundtruth/{}.txt'.format(i), mode="r",encoding='utf-16-le') as f:

for lines in f.readlines():

lines = lines.rstrip('\n')

if 'imdbID' in lines:

a2,b3,c2 = lines.partition(':')

c2 = c2.lstrip(' "')

c2 = c2.rstrip('",\n')

b2.append(c2+'.jpg')

if "Genre" in lines:

a,b1,c = lines.partition(':')

c = c.lstrip(' "')

c = c.rstrip('",\n')

c1 = c.split(',')

c1 = map(str.strip, c1)

b.append(list(c1))

f.close()

data = pd.DataFrame({'name':b2,'filename':b})

Now for the second step, we first append all the poster images filenames in the list.

q=[]
for i in range(1980,2016):
    for files in os.listdir('D:/downloads/Movie_Poster_Dataset/Movie_Poster_Dataset/{}'.format(i)):
        q.append(files)

q=[]

for i in range(1980,2016):

for files in os.listdir('D:/downloads/Movie_Poster_Dataset/Movie_Poster_Dataset/{}'.format(i)):

q.append(files)

Then check if the name is present in the dataframe or not. If not, we will remove the rows from the dataframe or create a new dataframe.

new = list(set(data['name']).intersection(q))
data2 = data[data['name'].isin(new)]

1 2	new = list(set(data['name']).intersection(q)) data2 = data[data['name'].isin(new)]

Be sure that we have no duplicates in the dataframe.

data2.drop_duplicates(subset= 'name',keep='first', inplace=True)

1	data2.drop_duplicates(subset= 'name',keep='first', inplace=True)

So, finally, we are ready with our cleaned dataset with 8052 images containing overall 25 classes. The dataframe is shown below.

One can also convert this dataframe into the common format as shown below

This can be done using the following code.

for idx, row in df.iterrows():
    for hobby in row.filename:
        df.loc[idx, hobby] = '1'

df.fillna('0', inplace=True)

for idx, row in df.iterrows():

for hobby in row.filename:

df.loc[idx, hobby] = '1'

df.fillna('0', inplace=True)

In this post, we will be using Format 1. You can use any. Here, we will be using the Keras flow_from_dataframe method. For this, we need to place all the images under one directory. Currently, all the images are in separate folders such as 1980, 1981, etc. Below is the code that places all the poster images in a single folder ‘original_train‘.

original_train = 'D:/downloads/Movie_Poster_Dataset/Data/'
for i in range(1980,2016):
    for files in os.listdir('D:/downloads/Movie_Poster_Dataset/Movie_Poster_Dataset/{}'.format(i)):
        src = os.path.join('D:/downloads/Movie_Poster_Dataset/Movie_Poster_Dataset/{}'.format(i),files)
        out = os.path.join(original_train,files)
        shutil.copy(src,out)

original_train = 'D:/downloads/Movie_Poster_Dataset/Data/'

for i in range(1980,2016):

for files in os.listdir('D:/downloads/Movie_Poster_Dataset/Movie_Poster_Dataset/{}'.format(i)):

src = os.path.join('D:/downloads/Movie_Poster_Dataset/Movie_Poster_Dataset/{}'.format(i),files)

out = os.path.join(original_train,files)

shutil.copy(src,out)

Model Architecture

model = Sequential()
model.add(Conv2D(16,(3,3),activation='relu',input_shape=(400,300,3)))
model.add(Dropout(0.25))
model.add(Conv2D(16,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Dropout(0.5))
model.add(Conv2D(32,(3,3),activation='relu'))
model.add(Conv2D(32,(3,3),activation='relu'))
model.add(Dropout(0.5))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Dropout(0.5))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(Conv2D(64,(1,1),activation='relu'))
model.add(MaxPool2D(pool_size=(4,4)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(25, activation='sigmoid'))

model = Sequential()

model.add(Conv2D(16,(3,3),activation='relu',input_shape=(400,300,3)))

model.add(Dropout(0.25))

model.add(Conv2D(16,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Dropout(0.5))

model.add(Conv2D(32,(3,3),activation='relu'))

model.add(Dropout(0.5))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(64,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Dropout(0.5))

model.add(Conv2D(128,(3,3),activation='relu'))

model.add(Conv2D(64,(1,1),activation='relu'))

model.add(MaxPool2D(pool_size=(4,4)))

model.add(Dropout(0.25))

model.add(Flatten())

model.add(Dense(512,activation='relu'))

model.add(Dropout(0.5))

model.add(Dense(25, activation='sigmoid'))

Since this is a sparse multilabel classification problem, accuracy is not a good metric for this. The reason for this is shown below.

if the predicted output was [0, 0, 0, 0, 0, 1] and the correct output was [0, 0, 0, 0, 0, 0], my accuracy would still be 5/6.

So, you can use other metrics like precision, recall, f1 score, hamming loss, top_k_categorical_accuracy, etc.

model.compile(optimizer='Adam',loss="binary_crossentropy",metrics=["accuracy",'top_k_categorical_accuracy'])

1	model.compile(optimizer='Adam',loss="binary_crossentropy",metrics=["accuracy",'top_k_categorical_accuracy'])

Here, I’ve used both to show how accuracy instantly reaches 90+ from the starting epoch and thus is not a correct metric.

flow_from_dataframe()

Here, I split the data into training and validation sets using the validation_split argument of ImageDataGenerator. You can read more about the ImageDataGenerator here.

datagen = ImageDataGenerator(rescale=1/255., validation_split=0.1)
 
train_generator = datagen.flow_from_dataframe(dataframe=data2, directory=original_train,
                                             x_col='name',
                                             y_col='filename',
                                             target_size=(400,300),
                                             color_mode='rgb',
                                             class_mode='categorical',
                                             batch_size=30,
                                             shuffle=True,
                                             subset='training',
                                             seed=7)
 
validation_generator = datagen.flow_from_dataframe(dataframe=data2, directory=original_train,
                                             x_col='name',
                                             y_col='filename',
                                             target_size=(400,300),
                                             color_mode='rgb',
                                             class_mode='categorical',
                                             batch_size=35,
                                             shuffle=False,
                                             subset='validation',
                                             seed=7)

datagen = ImageDataGenerator(rescale=1/255., validation_split=0.1)

train_generator = datagen.flow_from_dataframe(dataframe=data2, directory=original_train,

x_col='name',

y_col='filename',

target_size=(400,300),

color_mode='rgb',

class_mode='categorical',

batch_size=30,

shuffle=True,

subset='training',

seed=7)

validation_generator = datagen.flow_from_dataframe(dataframe=data2, directory=original_train,

x_col='name',

y_col='filename',

target_size=(400,300),

color_mode='rgb',

class_mode='categorical',

batch_size=35,

shuffle=False,

subset='validation',

seed=7)

Below are some of the poster images all resized into (400,300,3).

plt.figure(figsize=(10,5))
for i in range(6):
    plt.subplot(2,3,i+1)
    for x,y in train_generator:
        print(x.shape)
        plt.imshow(x[0])
        plt.xticks([])
        plt.yticks([])
        break
plt.tight_layout()
plt.show()

plt.figure(figsize=(10,5))

for i in range(6):

plt.subplot(2,3,i+1)

for x,y in train_generator:

print(x.shape)

plt.imshow(x[0])

plt.xticks([])

plt.yticks([])

break

plt.tight_layout()

plt.show()

You can also check which labels are assigned to which class using the following code.

train_generator.class_indices

1	train_generator.class_indices

This prints a dictionary containing class names as keys and labels as values.

Let’s start training…

train_steps = train_generator.n//train_generator.batch_size
validation_steps = validation_generator.n//validation_generator.batch_size
 
history = model.fit_generator(train_generator,steps_per_epoch=train_steps, epochs=10,
                              validation_data=validation_generator,validation_steps=validation_steps)

train_steps = train_generator.n//train_generator.batch_size

validation_steps = validation_generator.n//validation_generator.batch_size

history = model.fit_generator(train_generator,steps_per_epoch=train_steps, epochs=10,

validation_data=validation_generator,validation_steps=validation_steps)

See how accuracy is reaching 90+ within few epochs. As stated earlier this is not a good evaluation metric for multi-label classification. On the other hand, top_k_categorical_accuracy is showing us the true picture.

Clearly, we are doing a pretty decent job. Considering the fact that training data is small and the complexity of the problem is large(25 classes). Moreover, some classes like comedy, etc dominate the training data. Play with the model architecture and other hyperparameters and check how the accuracy varies.

Prediction time

For each image, let’s predict the top three predicted classes. Below is the code for this.

img = load_img('D:/downloads/Movie_Poster_Dataset/Data/tt0080854.jpg')
img1 = img_to_array(img)
img2 = cv2.resize(img1,(300,400))
img2 = img2/255
img3 = np.expand_dims(img2,axis=0)
proba = model.predict(img3)
top_3 = np.argsort(proba[0])[:-4:-1]
for i in range(3):
    print("{}".format(list(train_generator.class_indices.keys())[top_3[i]])+" ({:.3})".format(proba[0][top_3[i]]))
plt.imshow(img)

img = load_img('D:/downloads/Movie_Poster_Dataset/Data/tt0080854.jpg')

img1 = img_to_array(img)

img2 = cv2.resize(img1,(300,400))

img2 = img2/255

img3 = np.expand_dims(img2,axis=0)

proba = model.predict(img3)

top_3 = np.argsort(proba[0])[:-4:-1]

for i in range(3):

print("{}".format(list(train_generator.class_indices.keys())[top_3[i]])+" ({:.3})".format(proba[0][top_3[i]]))

plt.imshow(img)

The actual label for this can be found out as

data2[data2['name']=='tt0080854.jpg']

1	data2[data2['name']=='tt0080854.jpg']

You can see that our model is doing a decent job considering the complexity of the problem

Let’s try another example “tt0465602.jpg“. For this the predicted labels are

By looking at the poster most of us will predict the labels as predicted by our algorithm. Actually, these are pretty close to the true labels that are [Action, Comedy, Crime].

That’s all for multi-label classification problem. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – apply_transform method

Leave a reply

In this blog, we will discuss ImageDataGenerator “apply_transform” method. Using this method, you can apply any desired transformations to an image. You can find its use in the ImageDataGenerator “flow” method. First of all, let’s discuss its Keras API.

Keras API

apply_transform(x, transform_parameters)

1	apply_transform(x, transform_parameters)

This applies transformations to x (3D tensor) according to the transform parameters specified.

The “transform_parameters” is a dictionary specifying the set of transformations to be applied. Only the following transformations are available

theta, tx, ty, shear, zx, zy, flip_horizontal, flip_vertical, channel_shift_intensity, brightness

1	theta, tx, ty, shear, zx, zy, flip_horizontal, flip_vertical, channel_shift_intensity, brightness

Let’s discuss these in detail.

theta: Rotation angle in degrees. Below is an example that rotates the image by 40 degrees.

tx and ty: These are the shifts in the vertical and the horizontal directions respectively. For instance, tx=20 will shift the image vertically by 20 pixels.

In this, first of all, the translation matrix is calculated. Then affine transformation is applied using the “scipy.ndimage” affine_transformation method.

zx and zy: This zooms the image in the vertical and horizontal directions respectively. If less than 1, the image is zoomed in otherwise zoomed out.

Note: -ve values of zx and zy results in flipping the image in vertical and horizontal directions respectively. For instance, zx=-1 will flip the image vertically.

flip_horizontal and flip_vertical: This flips the image horizontally and vertically. For instance, below is the code for flipping the image horizontally,

b = img.swapaxes(1,0)
c = b[::-1, ...]
flip_img = c.swapaxes(0,1)

b = img.swapaxes(1,0)

c = b[::-1, ...]

flip_img = c.swapaxes(0,1)

channel_shift_intensity: This shifts the channel values by the amount specified. The following code sums up how it works

[np.clip(x_channel + amount, min_img, max_img) for x_channel in img]

1	[np.clip(x_channel + amount, min_img, max_img) for x_channel in img]

brightness: This controls the brightness of the image. An enhancement factor of 0.0 gives a black image. A factor of 1.0 gives the original image.

Hope you understand all the arguments. Now, let’s see how to use this.

How to use this?

Because “apply_transform” is a method inside the ImageDataGenerator class. Thus, one first need to create the instance of this class and then apply this method as shown below

from keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array
img = load_img('D:/downloads/opencv_logo.PNG')
img = img_to_array(img)
datagen = ImageDataGenerator()
rotate = datagen.apply_transform(x=img, transform_parameters={'theta':40, 'brightness':0.8, 'zx':0.9, 'zy':0.9})

from keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array

img = load_img('D:/downloads/opencv_logo.PNG')

img = img_to_array(img)

datagen = ImageDataGenerator()

rotate = datagen.apply_transform(x=img, transform_parameters={'theta':40, 'brightness':0.8, 'zx':0.9, 'zy':0.9})

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – get_random_transform method

Leave a reply

In the previous blog, we discussed how to generate batches of augmented data using the flow method. We also learned that the key ingredient in the flow method is the “get_random_transform” method. This generates random parameters for a transformation. So, in this blog, let’s discuss this method in detail.

Keras API

get_random_transform(img_shape, seed=None)

1	get_random_transform(img_shape, seed=None)

Now, let’s see how this generates random parameters for transformations by just using the image shape information.

How this works?

This borrows the parameters from the ImageDataGenerator class. For instance, if we define rotation range in the ImageDataGenerator class

datagen = ImageDataGenerator(rotation_range=40)

1	datagen = ImageDataGenerator(rotation_range=40)

then the random parameters for this is obtained as

if self.rotation_range:
    theta = np.random.uniform(-self.rotation_range,self.rotation_range)
else:
    theta = 0

if self.rotation_range:

theta = np.random.uniform(-self.rotation_range,self.rotation_range)

else:

theta = 0

Thus, whenever you generate examples, theta is obtained from the uniform distribution, specified according to the parameters provided in the ImageDataGenerator class.

Similarly, for every transformation provided in the ImageDataGenerator class, we can obtain the random parameters. For more details, refer to the Keras GitHub.

How to use this?

To use this, you first need to provide the transformations in the ImageDataGenerator class. For instance, if I just want to rotate the image, then first specify the parameters as

datagen = ImageDataGenerator(rotation_range=40)

1	datagen = ImageDataGenerator(rotation_range=40)

Now, to get the random parameters, call the “get_random_transform” method as

transform = datagen1.get_random_transform((257, 278, 3),seed = 2)

1	transform = datagen1.get_random_transform((257, 278, 3),seed = 2)

This outputs the following parameters dictionary as

See only the transformations specified in the ImageDataGenerator class i.e. theta value is changed. Rest all values are the default.

So, this way one can generate random parameters for transformations. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – standardize method

Leave a reply

In this blog, we will discuss ImageDataGenerator “standardize” method. This method performs in-place normalization to the batch of inputs. As already discussed, this is an important step in the flow method or data augmentation. So, let’s discuss it in detail.

Keras API

standardize(x)

1	standardize(x)

Here, x is the batch of inputs. This method returns the normalized inputs. Note that x is changed in-place. If you don’t want to change the inputs in-place, pass a copy of the input to this method.

How this works?

While performing data augmentation with ImageDataGenerator, we discussed different normalization techniques. These techniques include centering the entire distribution or a sample, rescaling the input, performing zca whitening, etc. Behind the scenes, these are implemented by the “standardize” method. Let’s see how.

For instance, we want to rescale the input by 1/255. So, first of all we will create an ImageDataGenerator instance as shown below

datagen = ImageDataGenerator(rescale=1/255.)

1	datagen = ImageDataGenerator(rescale=1/255.)

Then for data augmentation, we will use the flow method as

training_generator = datagen.flow(x_train, y_train, batch_size=64,subset='training',seed=7)

1	training_generator = datagen.flow(x_train, y_train, batch_size=64,subset='training',seed=7)

Thus, the training_generator will yield batches of augmented images. That’s all we usually do.

As already discussed in this blog, the flow method consists of three steps, of which the last step is the “standardize” method. All the normalization work in the ImageDataGenerator class is handled by this method.

Now, coming back to the above example, the “standardize” method will first check whether you want to rescale or not. If yes, then this will change the input in-place as shown below.

if self.rescale:
    x *= self.rescale

1 2	if self.rescale: x *= self.rescale

Similarly, this method performs featurewise_center or samplewise_center or any other normalization. For more details, refer to Keras Github.

How to use this?

First of all, create an ImageDataGenerator instance with the desired transformations. Then apply the “standardize” method as shown below.

import matplotlib.pyplot as plt
import numpy as np
from keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array

img = load_img('D:/downloads/opencv_logo.PNG')
img = img_to_array(img)
datagen2 = ImageDataGenerator(samplewise_center=True)
standardize = datagen2.standardize(np.copy(img))
plt.imshow(standardize/255.)

import matplotlib.pyplot as plt

import numpy as np

from keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array

img = load_img('D:/downloads/opencv_logo.PNG')

img = img_to_array(img)

datagen2 = ImageDataGenerator(samplewise_center=True)

standardize = datagen2.standardize(np.copy(img))

plt.imshow(standardize/255.)

Note: The standardize method only supports transformations that perform normalization such as featurewise_center, rescale, etc. Otherwise, this returns the same image or batch of inputs.

What does in-place means?

As already discussed, this method normalizes the inputs in-place. This is exactly what in-place operators in Python do. Let’s take an example to understand what does in-place means.

For instance, let’s rescale an image of all ones by 2. After the “standardize” method, see how the mean of the “images” change.

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(rescale=2)

images = np.ones((10, 128, 128, 3))
print(images.mean()) # 1.0 

images_std = datagen.standardize(images)

print(images.mean(), images_std.mean()) # 2.0 2.0

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(rescale=2)

images = np.ones((10, 128, 128, 3))

print(images.mean()) # 1.0

images_std = datagen.standardize(images)

print(images.mean(), images_std.mean()) # 2.0 2.0

That’s all for “standardize” method. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – random_transform method

Leave a reply

In the previous blog, we discussed how to generate random parameters for a transformation. In this blog, we will discuss how to apply a random transformation to an image.

Keras API

random_transform(x, seed=None)

1	random_transform(x, seed=None)

This function returns a randomly transformed version of the input image x.

How this method works?

First of all, this generates random parameters for a transformation using the “get_random_transform” method. For more details, refer to this blog.
Then the image is transformed according to the parameters (generated above) using the “apply_transform” method. For more details, refer to this blog.

Below is the code for this (taken from Keras)

def random_transform(self, x, seed=None):
    params = self.get_random_transform(x.shape, seed)
    return self.apply_transform(x, params)

def random_transform(self, x, seed=None):

params = self.get_random_transform(x.shape, seed)

return self.apply_transform(x, params)

How to use this?

To use this, you first need to provide the desired transformations in the ImageDataGenerator class. For instance, let’s say we just want to zoom the image. Firstly, we specify the parameters in the ImageDataGenerator class.

datagen = ImageDataGenerator(zoom_range=0.2)

1	datagen = ImageDataGenerator(zoom_range=0.2)

Then we apply the random_transform method.

random = datagen.random_transform(img)

1	random = datagen.random_transform(img)

Similarly, you can apply any random transformation to the image. Just specify the transformations in the ImageDataGenerator class. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – fit method

Leave a reply

In the previous blog, we discussed how to perform data augmentation using ImageDataGenerator. In that, we saw that some transformations require statistics of the entire dataset. These transformations include featurewise_center, featurewise_std_normalization and zca_whitening.

To calculate these statistics, first of all, one may need to load the entire dataset into the memory. Then calculate the mean, standard deviation, principal components or any other statistics from that data. Fortunately, Keras has a built-in fit method for doing this. Let’s discuss it in detail.

Keras API

fit(x, augment=False, rounds=1, seed=None)

1	fit(x, augment=False, rounds=1, seed=None)

Here, x is the data from which to calculate the statistics. Should have rank 4. Note that the channel axis of x should have value either 1, 3, or 4 depending upon whether the data is greyscale, RGB, or RGBA.

This also provides an option of whether to use the augmented data for calculating statistics or not. This is done using the “augment” argument. If True, then augmented examples are also used for calculating statistics. The number of augmented examples depends upon the “rounds” parameter. For instance, if “rounds=2” and x.shape[0] or data size is 64, then 128 augmented examples are used.

Below code shows its implementation (taken from Keras). First of all, create an array of zeros to handle the augmented examples. Then generate the augmented examples using the random_transform method and append to this array.

if augment:
    ax = np.zeros(
        tuple([rounds * x.shape[0]] + list(x.shape)[1:]),
        dtype=self.dtype)
    for r in range(rounds):
        for i in range(x.shape[0]):
            ax[i + r * x.shape[0]] = self.random_transform(x[i])
    x = ax

if augment:

ax = np.zeros(

tuple([rounds * x.shape[0]] + list(x.shape)[1:]),

dtype=self.dtype)

for r in range(rounds):

for i in range(x.shape[0]):

ax[i + r * x.shape[0]] = self.random_transform(x[i])

x = ax

Here, x is the training data or the data whose statistics we want to calculate. Once we have the data, we can easily calculate the statistics such as mean, standard deviation and principal components using Numpy and Scipy libraries.

Note: Statistics are calculated across all channels in an image. So, don’t calculate the mean separately for each channel.

Now, when we generate batches of augmented data using any method (like flow) these statistics are used to normalize the data as shown below

if self.featurewise_center:
    if self.mean is not None:
        x -= self.mean

if self.featurewise_center:

if self.mean is not None:

x -= self.mean

How to use this?

Let’s take the MNIST digit classification example. Suppose we want to center the distribution i.e. mean equal to 0. For this, we will use the ImageDataGenerator “featurewise_center” transformation. Firstly, load the data and preprocess it.

from keras.datasets import mnist
import numpy as np
from keras.preprocessing.image import ImageDataGenerator

(x_train, y_train),(x_test, y_test) = mnist.load_data()

train_x = x_train.astype('float32')
test_x = x_test.astype('float32')

train_x = np.expand_dims(train_x, axis=-1)
test_x = np.expand_dims(test_x, axis=-1)

from keras.datasets import mnist

import numpy as np

from keras.preprocessing.image import ImageDataGenerator

(x_train, y_train),(x_test, y_test) = mnist.load_data()

train_x = x_train.astype('float32')

test_x = x_test.astype('float32')

train_x = np.expand_dims(train_x, axis=-1)

test_x = np.expand_dims(test_x, axis=-1)

After loading the data, firstly, create an ImageDataGenerator instance. Then fit the training data as shown below

datagen = ImageDataGenerator(featurewise_center=True)
datagen.fit(train_x)

1 2	datagen = ImageDataGenerator(featurewise_center=True) datagen.fit(train_x)

Let’s calculate the mean of the training data manually and using “datagen” mean attribute.

print(train_x.mean())
print(datagen.mean)

1 2	print(train_x.mean()) print(datagen.mean)

As expected these should be the same i.e 33.318447. Now, let’s see what happens to the mean of the distribution after normalization.

datagenerator = datagen.flow(train_x,y_train,batch_size=len(train_x))
for x,y in datagenerator:
    print(x.mean())   # -2.752179e-05
    break

datagenerator = datagen.flow(train_x,y_train,batch_size=len(train_x))

for x,y in datagenerator:

print(x.mean()) # -2.752179e-05

break

Clearly, this centers the distribution. Similarly, we can perform other types of normalizations also.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras ImageDataGenerator Normalization at validation and test time

Leave a reply

Note: This blog should not be confused with Test time augmentation (TTA).

In the previous blogs, we discussed different operations that are available for image augmentation under the ImageDataGenerator class. For instance rotation, translation, zoom, shearing, normalization, etc. By this, our model will be exposed to more aspects of data and thus will generalize better.

But what about validation and prediction time? Since both of these are used to evaluate the model, we want them to be fixed. That is why we don’t apply any random transformation to the validation and test data. But the test and the dev sets should come from the same distribution as the train set. In other words, the test and the dev sets should be normalized using the statistics calculated on the train set.

Since the normalization in Keras is done using the ImageDataGenerator class. So, in this blog, we will discuss how to normalize the data during prediction using the ImageDataGenerator class?

Method-1

We create a separate ImageDataGenerator instance and then fit it on the train data as shown below.

train_datagen = ImageDataGenerator(featurewise_center=True,
                                  featurewise_std_normalization=True,
                                  rotation_range=40,
                                  width_shift_range=0.2,
                                  zoom_range=0.2,
                                  horizontal_flip=True)

# Fit the train_datagen to calculate the train data statistics
train_datagen.fit(x_train)

# Create a separate ImageDataGenerator instance
validation_datagen = ImageDataGenerator(featurewise_center=True,
                                        featurewise_std_normalization=True)

# Fit the validation_datagen on the train data 
validation_datagen.fit(x_train)

# Use any of the flow methods. Here I have used flow()
training_generator = train_datagen.flow(x_train, y_train, batch_size=64, seed=7)
validation_generator = validation_datagen.flow(x_val, y_val, batch_size=64, seed=7)

history = model.fit_generator(training_generator,steps_per_epoch=64, epochs=10, validation_data=validation_generator, validation_steps=32)

train_datagen = ImageDataGenerator(featurewise_center=True,

featurewise_std_normalization=True,

rotation_range=40,

width_shift_range=0.2,

zoom_range=0.2,

horizontal_flip=True)

# Fit the train_datagen to calculate the train data statistics

train_datagen.fit(x_train)

# Create a separate ImageDataGenerator instance

validation_datagen = ImageDataGenerator(featurewise_center=True,

featurewise_std_normalization=True)

# Fit the validation_datagen on the train data

validation_datagen.fit(x_train)

# Use any of the flow methods. Here I have used flow()

training_generator = train_datagen.flow(x_train, y_train, batch_size=64, seed=7)

validation_generator = validation_datagen.flow(x_val, y_val, batch_size=64, seed=7)

history = model.fit_generator(training_generator,steps_per_epoch=64, epochs=10, validation_data=validation_generator, validation_steps=32)

Similarly, we can do this for the test set. Because for validation and test set we need to fit the generator on the train data, this is very time-consuming.

Method-2

We use the “standardize” method provided under the ImageDataGenerator class. As already discussed, the “standardize” method performs in-place normalization to the batch of inputs, which makes it perfect for this work. You can read more about normalization here.

train_datagen = ImageDataGenerator(featurewise_center=True,
                                  featurewise_std_normalization=True,
                                  rotation_range=40,
                                  width_shift_range=0.2,
                                  zoom_range=0.2,
                                  horizontal_flip=True)

# Fit the train_datagen to calculate the train data statistics
train_datagen.fit(x_train)

# Apply standardize method
train_datagen.standardize(x_val)

training_generator = train_datagen.flow(x_train, y_train, batch_size=64, seed=7)

history = model.fit_generator(training_generator,steps_per_epoch=64, epochs=10, validation_data=(x_val,y_val))

train_datagen = ImageDataGenerator(featurewise_center=True,

featurewise_std_normalization=True,

rotation_range=40,

width_shift_range=0.2,

zoom_range=0.2,

horizontal_flip=True)

# Fit the train_datagen to calculate the train data statistics

train_datagen.fit(x_train)

# Apply standardize method

train_datagen.standardize(x_val)

training_generator = train_datagen.flow(x_train, y_train, batch_size=64, seed=7)

history = model.fit_generator(training_generator,steps_per_epoch=64, epochs=10, validation_data=(x_val,y_val))

Method-3

This is similar to the above method but is more explicit. In this we obtain the mean and the standard deviation from the generator and apply the desired normalization.

train_datagen = ImageDataGenerator(featurewise_center=True,
                                  featurewise_std_normalization=True,
                                  rotation_range=40,
                                  width_shift_range=0.2,
                                  zoom_range=0.2,
                                  horizontal_flip=True)

# Fit the train_datagen to calculate the train data statistics
train_datagen.fit(x_train)

# Apply the desired normalization
x_val -= train_datagen.mean
x_val /= train_datagen.std

training_generator = train_datagen.flow(x_train, y_train, batch_size=64, seed=7)

history = model.fit_generator(training_generator,steps_per_epoch=64, epochs=10, validation_data=(x_val,y_val))

train_datagen = ImageDataGenerator(featurewise_center=True,

featurewise_std_normalization=True,

rotation_range=40,

width_shift_range=0.2,

zoom_range=0.2,

horizontal_flip=True)

# Fit the train_datagen to calculate the train data statistics

train_datagen.fit(x_train)

# Apply the desired normalization

x_val -= train_datagen.mean

x_val /= train_datagen.std

training_generator = train_datagen.flow(x_train, y_train, batch_size=64, seed=7)

history = model.fit_generator(training_generator,steps_per_epoch=64, epochs=10, validation_data=(x_val,y_val))

I hope you might have now get some idea of how to apply normalization during prediction time. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Binary Classification

1 Reply

In this blog, we will learn how to perform binary classification using Convolution Neural Networks. Here, we will be using the classic dogs vs cats dataset, where we have to classify an image as belonging to one of these two classes. So, let’s get started.

Downloading the Dataset

This dataset was made available as a part of the Kaggle competition in 2013. You can download it from here. This dataset contains 25,000 labeled images of dogs and cats in the train folder and 12,500 unlabeled images in the test folder. The size of the images in the dataset is not the same. Some samples from the dataset are shown below

Since the competition is now closed, we can’t submit the test predictions to the kaggle. Thus, to know how good we are doing, we will make the test data from 25,000 labeled images.

Preparing the Data

Here, we will split the train folder into 20,000 for training, and 2500 each for validation and testing. For this, we will create 3 folders corresponding to each train, validation and test set. In these folders, we will create 2 sub-folders as cats and dogs. You can do this manually but here we will be using the Python os module. The code for creating folders and sub-folders is shown below

import os, shutil

# This is the path where you want to create these folders
original_path = 'D:/downloads/Data'

# Create 3 folders train, validation and test
train_dir = os.path.join(original_path,'train1')
os.mkdir(train_dir)
validation_dir = os.path.join(original_path,'validation1')
os.mkdir(validation_dir)
test_dir = os.path.join(original_path,'test1')
os.mkdir(test_dir)

# Create 2 sub-folders
train_cats_dir = os.path.join(train_dir,'cats')
os.mkdir(train_cats_dir)
train_dogs_dir = os.path.join(train_dir,'dogs')
os.mkdir(train_dogs_dir)

# Validation directory for cats and dogs
validation_cats_dir = os.path.join(validation_dir,'cats')
os.mkdir(validation_cats_dir)
validation_dogs_dir = os.path.join(validation_dir,'dogs')
os.mkdir(validation_dogs_dir)

# test directory for cats and dogs
test_cats_dir = os.path.join(test_dir,'cats')
os.mkdir(test_cats_dir)
test_dogs_dir = os.path.join(test_dir,'dogs')
os.mkdir(test_dogs_dir)

import os, shutil

# This is the path where you want to create these folders

original_path = 'D:/downloads/Data'

# Create 3 folders train, validation and test

train_dir = os.path.join(original_path,'train1')

os.mkdir(train_dir)

validation_dir = os.path.join(original_path,'validation1')

os.mkdir(validation_dir)

test_dir = os.path.join(original_path,'test1')

os.mkdir(test_dir)

# Create 2 sub-folders

train_cats_dir = os.path.join(train_dir,'cats')

os.mkdir(train_cats_dir)

train_dogs_dir = os.path.join(train_dir,'dogs')

os.mkdir(train_dogs_dir)

# Validation directory for cats and dogs

validation_cats_dir = os.path.join(validation_dir,'cats')

os.mkdir(validation_cats_dir)

validation_dogs_dir = os.path.join(validation_dir,'dogs')

os.mkdir(validation_dogs_dir)

# test directory for cats and dogs

test_cats_dir = os.path.join(test_dir,'cats')

os.mkdir(test_cats_dir)

test_dogs_dir = os.path.join(test_dir,'dogs')

os.mkdir(test_dogs_dir)

The above code will create folders and sub-folders in the original path specified above. Now, we will put the images in these folders. The below code places 20,000 images in train, 2500 each in the validation and test folder created above.

# This is the path where the downloaded train data is
original_train = os.path.join(original_path,'train')

# Put 20,000 images in train, 2500 each in the validation
# and test folder created above
for fname in os.listdir(original_train):
    category, index,_ = fname.split('.')
    index = int(index)
    src = os.path.join(original_train,fname)
    if category == 'cat':
        if index < 10000:
            out = os.path.join(train_cats_dir,fname)
            shutil.copy(src,out)
        elif index>=10000 and index<11250:
            out = os.path.join(validation_cats_dir)
            shutil.copy(src,out)
        elif index>=11250:
            out = os.path.join(test_cats_dir,fname)
            shutil.copy(src,out)
            
    else:
        if index < 10000:
            out = os.path.join(train_dogs_dir,fname)
            shutil.copy(src,out)
        elif index>=10000 and index<11250:
            out = os.path.join(validation_dogs_dir)
            shutil.copy(src,out)
        elif index>=11250:
            out = os.path.join(test_dogs_dir,fname)
            shutil.copy(src,out)

# This is the path where the downloaded train data is

original_train = os.path.join(original_path,'train')

# Put 20,000 images in train, 2500 each in the validation

# and test folder created above

for fname in os.listdir(original_train):

category, index,_ = fname.split('.')

index = int(index)

src = os.path.join(original_train,fname)

if category == 'cat':

if index < 10000:

out = os.path.join(train_cats_dir,fname)

shutil.copy(src,out)

elif index>=10000 and index<11250:

out = os.path.join(validation_cats_dir)

shutil.copy(src,out)

elif index>=11250:

out = os.path.join(test_cats_dir,fname)

shutil.copy(src,out)

else:

if index < 10000:

out = os.path.join(train_dogs_dir,fname)

shutil.copy(src,out)

elif index>=10000 and index<11250:

out = os.path.join(validation_dogs_dir)

shutil.copy(src,out)

elif index>=11250:

out = os.path.join(test_dogs_dir,fname)

shutil.copy(src,out)

Now, let’s display a sample image from say “train_cats_dir”. This is done using the following code.

import numpy as np
from keras.preprocessing.image import load_img
import matplotlib.pyplot as plt

fname = np.random.choice(os.listdir(train_cats_dir))
img = load_img(os.path.join(train_cats_dir, fname))
plt.imshow(img)

import numpy as np

from keras.preprocessing.image import load_img

import matplotlib.pyplot as plt

fname = np.random.choice(os.listdir(train_cats_dir))

img = load_img(os.path.join(train_cats_dir, fname))

plt.imshow(img)

Data Pre-processing

The data must be processed in an appropriate form before feeding in the neural network. This includes changing the data into numpy arrays, normalizing the values between 0 and 1 or any other suitable range, etc. This can be easily done using the keras ImageDataGenerator class. This is shown in the code below

from keras.preprocessing.image import ImageDataGenerator

# Normalizing the image values between 0 and 1
train_datagen = ImageDataGenerator(rescale=1./255)
validation_datagen = ImageDataGenerator(rescale=1./255)

from keras.preprocessing.image import ImageDataGenerator

# Normalizing the image values between 0 and 1

train_datagen = ImageDataGenerator(rescale=1./255)

validation_datagen = ImageDataGenerator(rescale=1./255)

Here, we will use the flow_from_directory method to generate batches of data.

train_generator = train_datagen.flow_from_directory(train_dir,target_size=(150,150),batch_size=100,class_mode='binary',interpolation='bilinear')
validation_generator = validation_datagen.flow_from_directory(validation_dir,target_size=(150,150),batch_size=100,class_mode='binary',interpolation='bilinear')

train_generator = train_datagen.flow_from_directory(train_dir,target_size=(150,150),batch_size=100,class_mode='binary',interpolation='bilinear')

validation_generator = validation_datagen.flow_from_directory(validation_dir,target_size=(150,150),batch_size=100,class_mode='binary',interpolation='bilinear')

Build Model

Since this is a binary classification problem, we use the sigmoid activation function in the last layer. The model architecture we will use is shown below.

from keras.layers import Dense, Flatten, Conv2D, MaxPool2D, Dropout
from keras.models import Sequential

model = Sequential()
model.add(Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Flatten())
model.add(Dense(512,activation='relu'))
model.add(Dense(1,activation='sigmoid'))

from keras.layers import Dense, Flatten, Conv2D, MaxPool2D, Dropout

from keras.models import Sequential

model = Sequential()

model.add(Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(64,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(128,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(128,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Flatten())

model.add(Dense(512,activation='relu'))

model.add(Dense(1,activation='sigmoid'))

model.summary()

1	model.summary()

For the compilation step, we will use the Adam optimizer with the binary crossentropy loss.

model.compile(loss='binary_crossentropy',optimizer = 'Adam',metrics=['accuracy'])

1	model.compile(loss='binary_crossentropy',optimizer = 'Adam',metrics=['accuracy'])

Callbacks

To have some control over the training, one must use callbacks. Here, we will be using ModelCheckpoint callback which save the model weights whenever the validation accuracy improves.

from keras.callbacks import ModelCheckpoint
filepath = 'D:/downloads/best_weights.hdf5'
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=0, save_best_only=True, save_weights_only=True, mode='auto',period=1)

from keras.callbacks import ModelCheckpoint

filepath = 'D:/downloads/best_weights.hdf5'

checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=0, save_best_only=True, save_weights_only=True, mode='auto',period=1)

Fit Model

history = model.fit_generator(train_generator, steps_per_epoch=200, epochs=20, validation_data=validation_generator, validation_steps=25,  callbacks=[checkpoint])

1	history = model.fit_generator(train_generator, steps_per_epoch=200, epochs=20, validation_data=validation_generator, validation_steps=25, callbacks=[checkpoint])

Visualize Training

Let’s visualize how the loss and accuracy vary during the training process. This is done using the History() object as shown below

train_acc = history.history['acc']
validation_acc = history.history['val_acc']
train_loss = history.history['loss']
validation_loss = history.history['val_loss']
epochs = range(1,21)
plt.plot(epochs,train_acc,'bo',label='Training Accuracy')
plt.plot(epochs,validation_acc,'b',label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend()
plt.figure()
plt.plot(epochs,train_loss,'bo',label='Training Loss')
plt.plot(epochs,validation_loss,'b',label='Validation Loss')
plt.title('Training and Validation Loss')
plt.legend()

train_acc = history.history['acc']

validation_acc = history.history['val_acc']

train_loss = history.history['loss']

validation_loss = history.history['val_loss']

epochs = range(1,21)

plt.plot(epochs,train_acc,'bo',label='Training Accuracy')

plt.plot(epochs,validation_acc,'b',label='Validation Accuracy')

plt.title('Training and Validation Accuracy')

plt.legend()

plt.figure()

plt.plot(epochs,train_loss,'bo',label='Training Loss')

plt.plot(epochs,validation_loss,'b',label='Validation Loss')

plt.title('Training and Validation Loss')

plt.legend()

Clearly, our model starts overfitting after 8 epoch. We know that to prevent overfitting, we can

Perform Data Augmentation
Use Dropout
Reduce the capacity of the network
Use Regularization etc.

So, let’s use Data Augmentation and Dropout and see how our model performs.

Data Augmentation

In this, we produce more examples from the existing examples by various operations such as rotating, translating, flipping, etc. Fortunately, in Keras, all these transformations can be performed using ImageDataGenerator class. Below is the code for this

train_datagen = ImageDataGenerator(rescale=1./255,
                                  rotation_range=40,
                                  width_shift_range=0.2,
                                  height_shift_range=0.2,
                                  shear_range=0.2,
                                  zoom_range=0.2,
                                  horizontal_flip=True)

train_datagen = ImageDataGenerator(rescale=1./255,

rotation_range=40,

width_shift_range=0.2,

height_shift_range=0.2,

shear_range=0.2,

zoom_range=0.2,

horizontal_flip=True)

Dropout

In this, we randomly turn off some neurons (setting to zero) during training. This can be easily implemented using the Keras Dropout layer. Here, I’ve just added a single Dropout layer before the Dense layer, with a dropout rate of 50%. Rest all the model is same as above.

model.add(Flatten())
model.add(Dropout(0.5))
model.add(Dense(512,activation='relu'))

model.add(Flatten())

model.add(Dropout(0.5))

model.add(Dense(512,activation='relu'))

Just change the train_datagen and add Dropout layer. Then, train the model the same as we did above. Let’s visualize the training process

Clearly, by looking at the plots, we are no longer overfitting. Just by adding a Dropout layer and augmentation, we have increased the accuracy from 87 to 95%. You can further improve the accuracy by using a pre-trained model and other regularization methods.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – flow_from_directory method

Leave a reply

In the previous blog, we learned how to generate batches of augmented data using the flow method. In that, the data was loaded in the memory. But this is not always the case. Sometimes, the datasets we download contains folders of data corresponding to the respective classes. To use the flow method, one may first need to append the data and corresponding labels into an array and then use the flow method on those arrays. Thus overall it is a tedious task.

This led to the need for a method that takes the path to a directory and generates batches of augmented data. In Keras, this is done using the flow_from_directory method. So, let’s discuss this method in detail.

Keras API

flow_from_directory(directory, target_size=(256, 256), color_mode='rgb', classes=None, class_mode='categorical', batch_size=32, shuffle=True, seed=None, save_to_dir=None, save_prefix='', save_format='png', follow_links=False, subset=None, interpolation='nearest')

1	flow_from_directory(directory, target_size=(256, 256), color_mode='rgb', classes=None, class_mode='categorical', batch_size=32, shuffle=True, seed=None, save_to_dir=None, save_prefix='', save_format='png', follow_links=False, subset=None, interpolation='nearest')

Here, the directory is the path of the directory that contains the sub-directories of the respective classes. Each subdirectory is treated as a different class. The name of the class can either be inferred from the subdirectory name or can be passed using the “classes” argument. The labels to these classes are assigned alphanumerically.

For instance, suppose you have a directory structure as shown below

data/
    train/
        dogs/
            dog001.jpg
            dog002.jpg
            ...
        cats/
            cat001.jpg
            cat002.jpg
            ...

data/

train/

dogs/

dog001.jpg

dog002.jpg

...

cats/

cat001.jpg

cat002.jpg

...

So, in this case, the directory will be the path to the train folder. If we set the “classes=None“, the class names will be inferred from the sub-directory names as “dogs” and “cats”. Because the labels are assigned alphanumerically, the labels for this will be {‘cats’: 0, ‘dogs’: 1}. If we have passed the argument classes=[‘Dog’,’Cat’], then the labels will be {‘Cat’: 0, ‘Dog’: 1}.

To check the class labels, we can use the “class_indices” argument as

train_datagen = ImageDataGenerator(...)
train_generator = train_datagen.flow_from_directory(...)
train_generator.class_indices

train_datagen = ImageDataGenerator(...)

train_generator = train_datagen.flow_from_directory(...)

train_generator.class_indices

This returns a dictionary containing the mapping from class names to class indices.

The labels generated depends on the “class_mode” argument. This can take one of “categorical“, “binary“, “sparse“, “input“, or None. Default is “categorical”.

If “binary“, the labels are “0” and “1”.
For “categorical“, we will have 2D one-hot encoded labels
If “sparse”, 1D integer labels
For autoencoders, pass this as “input“
Since during test time we have no labels, so pass as None.

Sometimes the datasets contain images that are not of the same size. So, using the “target_size” argument, we can resize the images to a fixed size using an interpolation method specified by the “interpolation” argument. Default is the nearest neighbor interpolation method.

You can also convert the color of the images using the “color_mode” argument. Available options are “grayscale“, “rgb“, “rgba“. Default is “rgb“.

You can also save the augmented images to the disk by specifying the “save_to_dir” argument. You can also select which format to save the image files and what prefix to use, using the “save_format” and “save_prefix” arguments respectively.

To see an example of flow_from_directory() method, you can refer to this blog.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

ImageDataGenerator – flow_from_dataframe method

Leave a reply

In the previous blogs, we discussed flow and flow_from_directory methods. Both these methods perform the same task i.e. generate batches of augmented data. The only thing that differs is the format or structuring of the datasets. Some of the most common formats (Image datasets) are

Keras builtin datasets
Datasets containing separate folders of data corresponding to the respective classes.
Datasets containing a single folder along with a CSV or JSON file that maps the image filenames with their corresponding classes.

We already know how to deal with the first two formats. In this blog, we will discuss how to perform data augmentation with the data available in the data frame. To do this, Keras provides a builtin flow_from_dataframe method. So, let’s discuss this method in detail.

Keras API

flow_from_dataframe(dataframe, directory=None, x_col='filename', y_col='class', target_size=(256, 256), color_mode='rgb', classes=None, class_mode='categorical', batch_size=32, shuffle=True, seed=None, save_to_dir=None, save_prefix='', save_format='png', subset=None, interpolation='nearest', drop_duplicates=True)

flow_from_dataframe(dataframe, directory=None, x_col='filename', y_col='class', target_size=(256, 256), color_mode='rgb', classes=None, class_mode='categorical', batch_size=32, shuffle=True, seed=None, save_to_dir=None, save_prefix='', save_format='png', subset=None, interpolation='nearest', drop_duplicates=True)

In this, you need to provide the data frame that contains the image names or file paths and the corresponding labels. Now, there are two cases possible:

if the data frame contains image names then you need to specify the directory where these images are residing, using the “directory” argument. See the example below.
if the data frame contains the absolute image paths then set the “directory” argument to None.

Similarly, for the labels column, the values can be string/list/tuple depending on the “class_mode” argument. For instance, if class_mode is binary, then the label column must contain the class values as strings. Note that we can have multiple label columns also. For instance regression tasks like bounding box prediction etc. Then you need to pass these columns as a list in the “y_col” argument.

Rest all the arguments are the same as discussed in the ImageDataGenerator flow_from_directory blog. Now let’s take an example to see how to use this.

We will take the traditional cats vs dogs dataset. First, download the dataset from Kaggle. This dataset contains two folders train and the test each containing 25000 and 12500 images respectively.

Create a Dataframe

The first step is to create a data frame that contains the filename and the corresponding labels column. For this, we will iterate over each image in the train folder and check the filename prefix. If it is a cat, set the label to 0 otherwise 1.

# Path to the train folder
original_train = 'D:/downloads/Data/train/'

filenames = os.listdir(original_train)
categories = []
for filename in filenames:
    category = filename.split('.')[0]
    if category == 'cat':
        categories.append('0')
    else:
        categories.append('1')

# Path to the train folder

original_train = 'D:/downloads/Data/train/'

filenames = os.listdir(original_train)

categories = []

for filename in filenames:

category = filename.split('.')[0]

if category == 'cat':

categories.append('0')

else:

categories.append('1')

Now create a data frame as

data = pd.DataFrame({'filename':filenames,'label':categories})

1	data = pd.DataFrame({'filename':filenames,'label':categories})

Create Generators

Now, we will create the train and validation generator using the flow_from_dataframe method as

datagen = ImageDataGenerator(rescale=1/255., validation_split=0.2)

train_generator = datagen.flow_from_dataframe(dataframe=data, directory=original_train,
                                             x_col='filename',
                                             y_col='label',
                                             target_size=(150,150),
                                             class_mode='binary',
                                             batch_size=100,
                                             subset='training',
                                             seed=7)

validation_generator = datagen.flow_from_dataframe(dataframe=data, directory=original_train,
                                             x_col='filename',
                                             y_col='label',
                                             target_size=(150,150),
                                             class_mode='binary',
                                             batch_size=100,
                                             subset='validation',
                                             seed=7)

datagen = ImageDataGenerator(rescale=1/255., validation_split=0.2)

train_generator = datagen.flow_from_dataframe(dataframe=data, directory=original_train,

x_col='filename',

y_col='label',

target_size=(150,150),

class_mode='binary',

batch_size=100,

subset='training',

seed=7)

validation_generator = datagen.flow_from_dataframe(dataframe=data, directory=original_train,

x_col='filename',

y_col='label',

target_size=(150,150),

class_mode='binary',

batch_size=100,

subset='validation',

seed=7)

Build the Model

model = Sequential()
model.add(Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPool2D((2,2)))
model.add(Flatten())
model.add(Dense(512,activation='relu'))
model.add(Dense(1,activation='sigmoid'))

model.compile(loss='binary_crossentropy',optimizer = 'Adam',metrics=['accuracy'])

model = Sequential()

model.add(Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(64,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(128,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Conv2D(128,(3,3),activation='relu'))

model.add(MaxPool2D((2,2)))

model.add(Flatten())

model.add(Dense(512,activation='relu'))

model.add(Dense(1,activation='sigmoid'))

model.compile(loss='binary_crossentropy',optimizer = 'Adam',metrics=['accuracy'])

Train the Model

Let’s train the model using the fit_generator method.

train_steps = train_generator.n//train_generator.batch_size
validation_steps = validation_generator.n//validation_generator.batch_size

history = model.fit_generator(train_generator,steps_per_epoch=train_steps, epochs=20,
                              validation_data=validation_generator,validation_steps=validation_steps)

train_steps = train_generator.n//train_generator.batch_size

validation_steps = validation_generator.n//validation_generator.batch_size

history = model.fit_generator(train_generator,steps_per_epoch=train_steps, epochs=20,

validation_data=validation_generator,validation_steps=validation_steps)

Test time

So, for the test time, we can simply use the flow_from_directory method. You can use any method. For this, you need to create a subfolder inside the test folder. Remember not to shuffle the data at the test time. The class_mode argument should be set to None.

test_directory = 'D:/downloads/Data/test/'
test_datagen = ImageDataGenerator(rescale=1/255.)
test_generator = test_datagen.flow_from_directory(test_directory,target_size=(150,150),
                                                 shuffle=False,
                                                 class_mode=None,
                                                 batch_size=1,
                                                 seed=7)

test_directory = 'D:/downloads/Data/test/'

test_datagen = ImageDataGenerator(rescale=1/255.)

test_generator = test_datagen.flow_from_directory(test_directory,target_size=(150,150),

shuffle=False,

class_mode=None,

batch_size=1,

seed=7)

For predictions, we can simply use the predict_generator method.

STEP_SIZE_TEST=test_generator.n//test_generator.batch_size
test_generator.reset()
pred=model.predict_generator(test_generator,steps=STEP_SIZE_TEST,verbose=1)

predictions = []
for i in pred:
    if i >=0.5:
        predictions.append('1')
    else:
        predictions.append('0')

filenames=test_generator.filenames
results=pd.DataFrame({"Filename":filenames,
                      "Predictions":predictions})

STEP_SIZE_TEST=test_generator.n//test_generator.batch_size

test_generator.reset()

pred=model.predict_generator(test_generator,steps=STEP_SIZE_TEST,verbose=1)

predictions = []

for i in pred:

if i >=0.5:

predictions.append('1')

else:

predictions.append('0')

filenames=test_generator.filenames

results=pd.DataFrame({"Filename":filenames,

"Predictions":predictions})

That’s all for the flow_from_dataframe method. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.