Category Archives: GAN

Implementation of GANs to generated Handwritten Digits

Leave a reply

In the previous blog, we studied about GANs, now in this blog, we will implement GANs to generate MNIST digits dataset.

In the generative adversarial networks, both generator and discriminator are trained simultaneously. Both networks can overpower each other if not trained properly. If discriminator is trained more than it will easily detect fake and real image then the generator will not able to generate real-looking images. And if the generator is trained heavily then discriminator will not be able to classify between real and fake images. We can solve this problem by properly setting the learning rate for both networks.

When we train discriminator we do not train generator and when we train generator we do not train discriminator. This makes the generator to train properly. Now, let’s look into the code for each part on the GAN network.

Discriminator Network:

We are using MNIST digits dataset which is having an image shape of (28, 28, 1). Since the image size is small we can use MLP network for discriminator instead of using convolutional layers. To do this first we need to reshape input into a single vector of size (784, 1). Then I have applied three dense layers of 512, 256 and 128 hidden units in each layers.

Generator Network:

To create generator network we will first take random noise as input with the shape of (100, 1). Then I have used three hidden layers with shape of 256, 512 and 1024. The output of the generator network is then reshaped to (28, 28, 1). I have batch normalization in each hidden layer. Batch normalization improves the quality of the trained model and also stabilizes the training process.

Combined Model:

To train the generator we need to create a combined model where we do not train the discriminator model. In combined model random noise is being given as input to the generator network and the output image is then passed through the discriminator network to get the label. Here I have flagged discriminator model as non-trainable.

Training the GAN network:

Training a GAN network requires careful hyper-parameters tuning. If the model is not trained carefully it will not converge to produce good results. We will use the following steps to train this GAN network:

  1. Firstly we will normalize input dataset (MNIST images).
  2. Train the discriminator with real images (from MNIST dataset)
  3. Sample same number of noise vectors to predict the output from generator network (Generator is not trained here).
  4. Train the discriminator network with images generated in the previous step.
  5. Take new random samples to train the generator with a combined model without training discriminator.
  6. Repeat from step 2-5 for some number of iterations. I have trained it for 30000 iterations.

Take a look into the generated images from this GAN network.

Here is the full code.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

An Introduction to Generative Adversarial Networks (GANs)

Leave a reply

Generative Adversarial Networks (GANs), as the name suggests these are the deep learning models used to generate new data from the given dataset using an adversarial process. GANs were first introduced by Ian Goodfellow at NIPS 2014. This idea is regarded as the most interesting in machine learning in the last 10 years. Generative models are carrying bigger hope because they can mimic any data distribution. They can be used to generated images, audio waveform containing speech, music, etc.

Generative Adversarial Network Algorithm:

To create a GAN, we train two networks simultaneously in an adversarial manner. The two networks are generator and discriminator. And the adversary is, while the generator tries to generate data similar to original data distribution, discriminator tries to discriminate between data generated by the generator and original data. Here generator will try to fool the discriminator by improving itself and discriminator tries to differentiate between original and fake. This training will continue until the discriminator model is fooled half the time and the generator is able to generate data similar to original data distribution.

Let’s consider an example of generating new images using GAN. The first network discriminator is D(X), where X is an image (either real or fake). And the second network generator is G(Z), where Z is random noise. To train these networks D is first fed with real images and train to produce values close to 1(real) and then fed with fake images(generated by generator) and trained to produce values close to 0 (fake). Similarly, the generator is trained with loss generated by each image fed to discriminator produced by the generator.

We train D to maximize the probability of assigning the correct label to both training examples and samples from G. We simultaneously train G to minimize log(1 − D(G(z))). Let’s take a look into the algorithm provided in GAN paper.

Paper

We train this network for some number of iterations to make generator predict images close to the training dataset.

Generative Adversarial Networks (GANs) Vs Variational Autoencoders (VAEs)

There are some other generative models such as variational autoencoders that can do a similar job as GANs do. A VAE model maps the input to low dimensional space and then create a probability distribution to generate new outputs using some decoder function (To know more about VAEs you can follow this blog).

VAE Model

While Vanilla GANs are not able to map the input to latent space rather they use random noise to generate new data. GANs are usually difficult to train but generate more fine and granular images while VAEs are easier to train but produces more blurred images.

This was a brief introduction about generative adversarial networks. In the following posts, we will implement different GAN architectures, train GAN network and learn more about GAN improvements with its variants (CycleGAN, InfoGAN, BigGAN, etc).

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Implementing semi-supervised Learning using GANs

Leave a reply

Semi-supervised learning aims to make use of a large amount of unlabelled data to boost the performance of a model having less amount of labeled data. These type of models can be very useful when collecting labeled data is quite cumbersome and expensive. Several semi-supervised deep learning models have performed quite well on standard benchmarks. In this blog, we will learn how GANs can help in semi-supervised learning.

If you are new to GANs, you should first read this blog: An Introduction to Generative Adversarial Networks. Generally in GANs, we train using two networks adversely, generator and discriminator. After training the GAN network we discard the discriminator and only use generator network to generate the new data. Now in the semi-supervised model after training the network we will discard the generator model and use the discriminator model. But here the discriminator model is designed differently.

In semi-supervised GAN (SGAN) discriminator is not only trained to discriminate between real and fake data but also to predict the label for the input image. Let say we take an example of MNIST dataset. In MNIST dataset there are basically handwritten digits from 0-9, a total of 10 classes. Now in semi-supervised GAN for MNIST digits, the discriminator will be trained for real or fake images and for predicting these 10 classes also.

So in SGANs, the discriminator is trained with these three types of datasets.

  1. Fake images generated by generator network.
  2. Real images from a dataset without having any labels (a large amount of unlabeled data).
  3. Real images from the dataset with labels ( less number of the labeled dataset)

While generator in SGAN will be trained in a similar way as it is trained in vanilla GANs. This type of training will allow the model to learn useful features extracted from unlabeled dataset and use these features to train a supervised discriminator to predict the labels of the input image.

Implementing Semi-Supervised GAN

Now we will implement a semi-supervised GAN using MNIST digits dataset. If you want to implement a simple GAN you can follow this blog: Implementation of GANs to generated Handwritten Digits.

MNIST digits dataset consists of 60000 training images from which we will only use 1000 labeled images and rest as unlabeled images. We will select random 1000 labeled images containing 100 images for each class. Let’s see the code for this:

Discriminator in SGAN

For this semi-supervised GAN model, we will create two discriminator models both of them share weights of every layer but have different output layers. One model will be the binary classifier model (discriminate between real and fake images) and another will be multi-class classifier model (predicts labels for the input image). Let’s see the code for this:

Generator in SGAN

Generator in this SGAN is a simple multi-layer neural network having three hidden layers with units 512, 256 and 128. The output layer is having a shape of the original image (28, 28,1). Input to the generator will we random noise of vector size 100. Here is the code.

Training the model

Training this model will consist of the following steps:

  1. Sample both label and unlabeled data from the MNIST dataset, also normalize and make labels of data into categorical form.
  2. Train the multi-class discriminator model with labeled real images (take a batch from images)
  3. Train the binary-class discriminator model with unlabeled real images (take a batch from images)
  4. Sample noise of vector size 100 and train the binary-class discriminator model with fake images generated by generator network.
  5. Sample noise of vector size 100 and train the combined model to train the generator network.
  6. Repeat steps from 2-5 for some number of iterations. I have trained it for 10000 iterations.

In the above training steps, you can see that we are training multi-class discriminator and binary-class discriminator in different steps. But actually they are sharing weights of the same network except for the output layer (As I have mentioned earlier).

Also, Binary-class discriminator is trained two times in every iteration, one with real images taken from the dataset and another with fake images generated from the generative network. While multi-class discriminator is trained once in each iteration, only with real labeled images. This is because multi-class labels are not available for generated images.

I have also tested the SGAN model with 10000 test dataset provided by MNIST after every 1000 iteration. Here is the result of that.

Now you can see that I have trained this SGAN model with only 1000 labeled images and it gives an accuracy of about 94.8%, that is quite nice.

Give me the full code!

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.