Tag Archives: generative model

Implementation of GANs to generated Handwritten Digits

In the previous blog, we studied about GANs, now in this blog, we will implement GANs to generate MNIST digits dataset.

In the generative adversarial networks, both generator and discriminator are trained simultaneously. Both networks can overpower each other if not trained properly. If discriminator is trained more than it will easily detect fake and real image then the generator will not able to generate real-looking images. And if the generator is trained heavily then discriminator will not be able to classify between real and fake images. We can solve this problem by properly setting the learning rate for both networks.

When we train discriminator we do not train generator and when we train generator we do not train discriminator. This makes the generator to train properly. Now, let’s look into the code for each part on the GAN network.

Discriminator Network:

We are using MNIST digits dataset which is having an image shape of (28, 28, 1). Since the image size is small we can use MLP network for discriminator instead of using convolutional layers. To do this first we need to reshape input into a single vector of size (784, 1). Then I have applied three dense layers of 512, 256 and 128 hidden units in each layers.

def discriminator(self):

    input_disc = Input(shape = (784,))
    hidden1 = Dense(512, activation = 'relu')(input_disc)
    hidden2 = Dense(256, activation = 'relu')(hidden1)
    hidden3 = Dense(128, activation = 'relu')(hidden2)
    output = Dense(1, activation = 'sigmoid')(hidden3)
    disc_model = Model(input_disc, output)
    disc_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])
    print(disc_model.summary())

    return disc_model

def discriminator(self):

input_disc = Input(shape = (784,))

hidden1 = Dense(512, activation = 'relu')(input_disc)

hidden2 = Dense(256, activation = 'relu')(hidden1)

hidden3 = Dense(128, activation = 'relu')(hidden2)

output = Dense(1, activation = 'sigmoid')(hidden3)

disc_model = Model(input_disc, output)

disc_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])

print(disc_model.summary())

return disc_model

Generator Network:

To create generator network we will first take random noise as input with the shape of (100, 1). Then I have used three hidden layers with shape of 256, 512 and 1024. The output of the generator network is then reshaped to (28, 28, 1). I have batch normalization in each hidden layer. Batch normalization improves the quality of the trained model and also stabilizes the training process.

def generator(self):

    input_gen = Input(shape = (self.latent_dim,))
    hidden1 = BatchNormalization(momentum=0.8)(Dense(256, activation = 'relu')(input_gen))
    hidden2 = BatchNormalization(momentum=0.8)(Dense(512, activation = 'relu')(hidden1))
    hidden3 = BatchNormalization(momentum=0.8)(Dense(1024, activation = 'relu')(hidden2))
    output = Dense(784, activation='tanh')(hidden3)
    reshaped_output = Reshape((28, 28, 1))(output)
    gen_model = Model(input_gen, reshaped_output)
    gen_model.compile(loss='binary_crossentropy', optimizer=self.optimizer)
    print(gen_model.summary())


    return gen_model

def generator(self):

input_gen = Input(shape = (self.latent_dim,))

hidden1 = BatchNormalization(momentum=0.8)(Dense(256, activation = 'relu')(input_gen))

hidden2 = BatchNormalization(momentum=0.8)(Dense(512, activation = 'relu')(hidden1))

hidden3 = BatchNormalization(momentum=0.8)(Dense(1024, activation = 'relu')(hidden2))

output = Dense(784, activation='tanh')(hidden3)

reshaped_output = Reshape((28, 28, 1))(output)

gen_model = Model(input_gen, reshaped_output)

gen_model.compile(loss='binary_crossentropy', optimizer=self.optimizer)

print(gen_model.summary())

return gen_model

Combined Model:

To train the generator we need to create a combined model where we do not train the discriminator model. In combined model random noise is being given as input to the generator network and the output image is then passed through the discriminator network to get the label. Here I have flagged discriminator model as non-trainable.

def combined(self):

    inputs = Input(shape = (self.latent_dim,)) 
    gen_img = self.generator_model(inputs)
    gen_img = Reshape((784,))(gen_img)
    self.discriminator_model.trainable = False
    outs = self.discriminator_model(gen_img)
    comb_model = Model(inputs, outs)
    comb_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])
    print(comb_model.summary())

    return comb_model

def combined(self):

inputs = Input(shape = (self.latent_dim,))

gen_img = self.generator_model(inputs)

gen_img = Reshape((784,))(gen_img)

self.discriminator_model.trainable = False

outs = self.discriminator_model(gen_img)

comb_model = Model(inputs, outs)

comb_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])

print(comb_model.summary())

return comb_model

Training the GAN network:

Training a GAN network requires careful hyper-parameters tuning. If the model is not trained carefully it will not converge to produce good results. We will use the following steps to train this GAN network:

Firstly we will normalize input dataset (MNIST images).
Train the discriminator with real images (from MNIST dataset)
Sample same number of noise vectors to predict the output from generator network (Generator is not trained here).
Train the discriminator network with images generated in the previous step.
Take new random samples to train the generator with a combined model without training discriminator.
Repeat from step 2-5 for some number of iterations. I have trained it for 30000 iterations.

def train(self):

    train_data = (self.x_train.astype(np.float32) - 127.5) / 127.5

    for i in range(self.iterations):

        batch_indx = np.random.randint(0, train_data.shape[0], size = (self.half_batch_size))
        batch_x = train_data[batch_indx]
        batch_x = batch_x.reshape((-1, 784))

        input_noise = np.random.normal(0, 1, size=(self.half_batch_size, 100))
        gen_outs = self.generator_model.predict(input_noise)
        gen_outs = gen_outs.reshape((-1, 784))

        real_loss = self.discriminator_model.train_on_batch(batch_x, np.ones((self.half_batch_size,1)))
        fake_loss = self.discriminator_model.train_on_batch(gen_outs, np.zeros((self.half_batch_size,1)))

        disc_loss = 0.5*np.add(fake_loss,real_loss)

        full_batch_input_noise = np.random.normal(0, 1, size=(self.batch_size, 100))
        gan_loss = self.combined_model.train_on_batch(full_batch_input_noise, np.array([1] * self.batch_size))

        print(i, disc_loss, gan_loss)

def train(self):

train_data = (self.x_train.astype(np.float32) - 127.5) / 127.5

for i in range(self.iterations):

batch_indx = np.random.randint(0, train_data.shape[0], size = (self.half_batch_size))

batch_x = train_data[batch_indx]

batch_x = batch_x.reshape((-1, 784))

input_noise = np.random.normal(0, 1, size=(self.half_batch_size, 100))

gen_outs = self.generator_model.predict(input_noise)

gen_outs = gen_outs.reshape((-1, 784))

real_loss = self.discriminator_model.train_on_batch(batch_x, np.ones((self.half_batch_size,1)))

fake_loss = self.discriminator_model.train_on_batch(gen_outs, np.zeros((self.half_batch_size,1)))

disc_loss = 0.5*np.add(fake_loss,real_loss)

full_batch_input_noise = np.random.normal(0, 1, size=(self.batch_size, 100))

gan_loss = self.combined_model.train_on_batch(full_batch_input_noise, np.array([1] * self.batch_size))

print(i, disc_loss, gan_loss)

Take a look into the generated images from this GAN network.

Here is the full code.

from keras.layers import Input, Dense, Reshape, BatchNormalization
from keras.models import Model
from keras.optimizers import Adam
from keras.datasets import mnist
import numpy as np

class GAN():
    def __init__(self):

        (self.x_train, self.y_train), (self.x_test, self.y_test) = mnist.load_data()
        self.batch_size = 128
        self.half_batch_size = 64
        self.latent_dim = 100
        self.iterations = 30000
        self.optimizer = Adam(0.0002, 0.5)
        self.generator_model = self.generator() 
        self.discriminator_model = self.discriminator()
        self.combined_model = self.combined()
        

    def generator(self):
        
        input_gen = Input(shape = (self.latent_dim,))
        hidden1 = BatchNormalization(momentum=0.8)(Dense(256, activation = 'relu')(input_gen))
        hidden2 = BatchNormalization(momentum=0.8)(Dense(512, activation = 'relu')(hidden1))
        hidden3 = BatchNormalization(momentum=0.8)(Dense(1024, activation = 'relu')(hidden2))
        output = Dense(784, activation='tanh')(hidden3)
        reshaped_output = Reshape((28, 28, 1))(output)
        gen_model = Model(input_gen, reshaped_output)
        gen_model.compile(loss='binary_crossentropy', optimizer=self.optimizer)
        print(gen_model.summary())
        
        
        return gen_model
    
    def discriminator(self):
        
        input_disc = Input(shape = (784,))
        hidden1 = Dense(512, activation = 'relu')(input_disc)
        hidden2 = Dense(256, activation = 'relu')(hidden1)
        hidden3 = Dense(128, activation = 'relu')(hidden2)
        output = Dense(1, activation = 'sigmoid')(hidden3)
        disc_model = Model(input_disc, output)
        disc_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])
        print(disc_model.summary())
        
        return disc_model
    
    def combined(self):
        
        inputs = Input(shape = (self.latent_dim,)) 
        gen_img = self.generator_model(inputs)
        gen_img = Reshape((784,))(gen_img)
        self.discriminator_model.trainable = False
        outs = self.discriminator_model(gen_img)
        comb_model = Model(inputs, outs)
        comb_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])
        print(comb_model.summary())
        
        return comb_model
    
    def train(self):
        
        train_data = (self.x_train.astype(np.float32) - 127.5) / 127.5
        
        for i in range(self.iterations):
            
            batch_indx = np.random.randint(0, train_data.shape[0], size = (self.half_batch_size))
            batch_x = train_data[batch_indx]
            batch_x = batch_x.reshape((-1, 784))
            
            input_noise = np.random.normal(0, 1, size=(self.half_batch_size, 100))
            gen_outs = self.generator_model.predict(input_noise)
            gen_outs = gen_outs.reshape((-1, 784))
            
            fake_loss = self.discriminator_model.train_on_batch(gen_outs, np.zeros((self.half_batch_size,1)))
            real_loss = self.discriminator_model.train_on_batch(batch_x, np.ones((self.half_batch_size,1)))
            
            disc_loss = 0.5*np.add(fake_loss,real_loss)
            
            full_batch_input_noise = np.random.normal(0, 1, size=(self.batch_size, 100))
            gan_loss = self.combined_model.train_on_batch(full_batch_input_noise, np.array([1] * self.batch_size))
            
            print(i, disc_loss, gan_loss)
            
# training the network
gan = GAN()
gan.train()


# generating new images from trained network
import matplotlib.pyplot as plt

r, c = 5, 5
noise = np.random.normal(0, 1, (r * c, 100))

gen_imgs = gan.generator_model.predict(noise)

# Rescale images 0 - 1
gen_imgs = 0.5 * gen_imgs + 0.5

fig, axs = plt.subplots(r, c)
cnt = 0
for i in range(r):
    for j in range(c):
        axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')
        axs[i,j].axis('off')
        cnt += 1
        
plt.show()
fig.savefig("mnist.png")
plt.close()

100

101

102

103

104

105

106

107

108

109

110

111

112

from keras.layers import Input, Dense, Reshape, BatchNormalization

from keras.models import Model

from keras.optimizers import Adam

from keras.datasets import mnist

import numpy as np

class GAN():

def __init__(self):

(self.x_train, self.y_train), (self.x_test, self.y_test) = mnist.load_data()

self.batch_size = 128

self.half_batch_size = 64

self.latent_dim = 100

self.iterations = 30000

self.optimizer = Adam(0.0002, 0.5)

self.generator_model = self.generator()

self.discriminator_model = self.discriminator()

self.combined_model = self.combined()

def generator(self):

input_gen = Input(shape = (self.latent_dim,))

hidden1 = BatchNormalization(momentum=0.8)(Dense(256, activation = 'relu')(input_gen))

hidden2 = BatchNormalization(momentum=0.8)(Dense(512, activation = 'relu')(hidden1))

hidden3 = BatchNormalization(momentum=0.8)(Dense(1024, activation = 'relu')(hidden2))

output = Dense(784, activation='tanh')(hidden3)

reshaped_output = Reshape((28, 28, 1))(output)

gen_model = Model(input_gen, reshaped_output)

gen_model.compile(loss='binary_crossentropy', optimizer=self.optimizer)

print(gen_model.summary())

return gen_model

def discriminator(self):

input_disc = Input(shape = (784,))

hidden1 = Dense(512, activation = 'relu')(input_disc)

hidden2 = Dense(256, activation = 'relu')(hidden1)

hidden3 = Dense(128, activation = 'relu')(hidden2)

output = Dense(1, activation = 'sigmoid')(hidden3)

disc_model = Model(input_disc, output)

disc_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])

print(disc_model.summary())

return disc_model

def combined(self):

inputs = Input(shape = (self.latent_dim,))

gen_img = self.generator_model(inputs)

gen_img = Reshape((784,))(gen_img)

self.discriminator_model.trainable = False

outs = self.discriminator_model(gen_img)

comb_model = Model(inputs, outs)

comb_model.compile(loss='binary_crossentropy', optimizer=self.optimizer, metrics=['accuracy'])

print(comb_model.summary())

return comb_model

def train(self):

train_data = (self.x_train.astype(np.float32) - 127.5) / 127.5

for i in range(self.iterations):

batch_indx = np.random.randint(0, train_data.shape[0], size = (self.half_batch_size))

batch_x = train_data[batch_indx]

batch_x = batch_x.reshape((-1, 784))

input_noise = np.random.normal(0, 1, size=(self.half_batch_size, 100))

gen_outs = self.generator_model.predict(input_noise)

gen_outs = gen_outs.reshape((-1, 784))

fake_loss = self.discriminator_model.train_on_batch(gen_outs, np.zeros((self.half_batch_size,1)))

real_loss = self.discriminator_model.train_on_batch(batch_x, np.ones((self.half_batch_size,1)))

disc_loss = 0.5*np.add(fake_loss,real_loss)

full_batch_input_noise = np.random.normal(0, 1, size=(self.batch_size, 100))

gan_loss = self.combined_model.train_on_batch(full_batch_input_noise, np.array([1] * self.batch_size))

print(i, disc_loss, gan_loss)

# training the network

gan = GAN()

gan.train()

# generating new images from trained network

import matplotlib.pyplot as plt

r, c = 5, 5

noise = np.random.normal(0, 1, (r * c, 100))

gen_imgs = gan.generator_model.predict(noise)

# Rescale images 0 - 1

gen_imgs = 0.5 * gen_imgs + 0.5

fig, axs = plt.subplots(r, c)

cnt = 0

for i in range(r):

for j in range(c):

axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')

axs[i,j].axis('off')

cnt += 1

plt.show()

fig.savefig("mnist.png")

plt.close()

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Variational Autoencoders

Variational Autoencoder Model

A variational autoencoder has encoder and decoder part mostly same as autoencoders, the difference is instead of creating a compact distribution from its encoder, it learns a latent variable model. These latent variables are used to create a probability distribution from which input for the decoder is generated. Another is, instead of using mean squared or cross entropy loss function (as in autoencoders ) it has its own loss function.

I will not go further into the mathematics behind it, Lets jump into the code which will give more understanding about variational autoencoders. To know more about the mathematics behind it please go through this tutorial.

I have implemented variational autoencoder in keras using MNIST dataset. So lets first download the data.

# download training and test data from mnist and reshape it

from keras.datasets import mnist
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
X_train = X_train.astype('float32') / 255.
X_train = X_train.reshape(-1,28,28,1)

X_test = X_test.astype('float32') / 255.
X_test = X_test.reshape(-1,28,28,1)
print(X_train.shape, X_test.shape)

# download training and test data from mnist and reshape it

from keras.datasets import mnist

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.astype('float32') / 255.

X_train = X_train.reshape(-1,28,28,1)

X_test = X_test.astype('float32') / 255.

X_test = X_test.reshape(-1,28,28,1)

print(X_train.shape, X_test.shape)

Now create an encoder model as it is created in autoencoders.

# Create encoder network

inputs = Input(shape = (28,28,1))
conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(inputs)
conv1_1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(conv1)
pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1_1)
conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1)
pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2)

flat = Flatten()(pool2)
input_to_z = Dense(32, activation = 'relu')(flat)

# Create encoder network

inputs = Input(shape = (28,28,1))

conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(inputs)

conv1_1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(conv1)

pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1_1)

conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1)

pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2)

flat = Flatten()(pool2)

input_to_z = Dense(32, activation = 'relu')(flat)

Latent Distribution Parameters and Function

Now encode the output of the encoder to latent distribution parameters. Here, I have created two parameters mu and sigma which represents the mean and standard distribution of the distribution.

latent_dim = 2 # dimension of latent variable
mu = Dense(latent_dim, name='mu')(input_to_z)
sigma = Dense(latent_dim, name='log_var')(input_to_z)

encoder = Model(inputs, mu)

latent_dim = 2 # dimension of latent variable

mu = Dense(latent_dim, name='mu')(input_to_z)

sigma = Dense(latent_dim, name='log_var')(input_to_z)

encoder = Model(inputs, mu)

Here I have taken latent space dimension equal to 2. This is the bottleneck which means we are passing our entire set of data to two single variables. So if we increase our latent space dimension to 5, 10 or higher, we can get better results in the output. But this will create more data in the bottleneck.

Now create a Gaussian distribution function with mean zero and standard deviation of 1. This distribution will give variation in the input to the decoder, which will help to get variation in the output. Then decoder will predict the output using distribution.

# create latent distribution function and generate vectors

def sampling(args):
    mu, sigma = args
    epsilon = K.random_normal(shape=(K.shape(mu)[0], latent_dim),
                              mean=0., stddev=1.)
    return mu + K.exp(sigma) * epsilon

z = Lambda(sampling)([mu, sigma])

#create decoder network which is reverse of encoder

decoder_inputs = Input(K.int_shape(z)[1:])
dense_layer_d = Dense(7*7*32, activation = 'relu')(decoder_inputs)
output_from_z_d = Reshape((7,7,32))(dense_layer_d)
trans1_d = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(output_from_z_d)
trans1_1_d = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(trans1_d)
trans2_d = Conv2DTranspose(1, 3, padding='same', activation='relu')(trans1_1_d)


decoder = Model(decoder_inputs, trans2_d)
z_decoded = decoder(z)

# create latent distribution function and generate vectors

def sampling(args):

mu, sigma = args

epsilon = K.random_normal(shape=(K.shape(mu)[0], latent_dim),

mean=0., stddev=1.)

return mu + K.exp(sigma) * epsilon

z = Lambda(sampling)([mu, sigma])

#create decoder network which is reverse of encoder

decoder_inputs = Input(K.int_shape(z)[1:])

dense_layer_d = Dense(7*7*32, activation = 'relu')(decoder_inputs)

output_from_z_d = Reshape((7,7,32))(dense_layer_d)

trans1_d = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(output_from_z_d)

trans1_1_d = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(trans1_d)

trans2_d = Conv2DTranspose(1, 3, padding='same', activation='relu')(trans1_1_d)

decoder = Model(decoder_inputs, trans2_d)

z_decoded = decoder(z)

Loss Function

For the loss function, a variational autoencoder uses the sum of two losses, one is the generative loss which is a binary cross entropy loss and measures how accurately the image is predicted, another is the latent loss, which is KL divergence loss, measures how closely a latent variable match Gaussian distribution. This KL divergence makes sure that our distribution generated from encoder do not go away from the origin. Then train the model.

#calculate reconstruction loss and KL divergence

class calc_output_with_los(keras.layers.Layer):

    def vae_loss(self, x, z_decoded):
        x = K.flatten(x)
        z_decoded = K.flatten(z_decoded)

        xent_loss = keras.metrics.binary_crossentropy(x, z_decoded)

        kl_loss = -5e-4 * K.mean(1 + sigma - K.square(mu) - K.exp(sigma), axis=-1)
        return K.mean(xent_loss + kl_loss)

    def call(self, inputs):
        x = inputs[0]
        z_decoded = inputs[1]
        loss = self.vae_loss(x, z_decoded)
        self.add_loss(loss, inputs=inputs)
        return x

outputs = calc_output_with_los()([inputs, z_decoded])

# define variational autoencoder model and train it

vae = Model(inputs, outputs)
m = 256
n_epoch = 10
vae.compile(optimizer='adam', loss=None)
vae.fit(X_train, epochs=n_epoch, batch_size=m, shuffle=True, validation_data=(X_test, None))

#calculate reconstruction loss and KL divergence

class calc_output_with_los(keras.layers.Layer):

def vae_loss(self, x, z_decoded):

x = K.flatten(x)

z_decoded = K.flatten(z_decoded)

xent_loss = keras.metrics.binary_crossentropy(x, z_decoded)

kl_loss = -5e-4 * K.mean(1 + sigma - K.square(mu) - K.exp(sigma), axis=-1)

return K.mean(xent_loss + kl_loss)

def call(self, inputs):

x = inputs[0]

z_decoded = inputs[1]

loss = self.vae_loss(x, z_decoded)

self.add_loss(loss, inputs=inputs)

return x

outputs = calc_output_with_los()([inputs, z_decoded])

# define variational autoencoder model and train it

vae = Model(inputs, outputs)

m = 256

n_epoch = 10

vae.compile(optimizer='adam', loss=None)

vae.fit(X_train, epochs=n_epoch, batch_size=m, shuffle=True, validation_data=(X_test, None))

Our model is ready and we can generate images from it very easily. All we need to do is sample latent variable from distribution and pass it to the decoder. Lets test with the following code:

n = 15  # figure with 15x15 digits

digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))

grid_x = np.linspace(-1, 1, n)
grid_y = np.linspace(-1, 1, n)

for i, yi in enumerate(grid_x):
    for j, xi in enumerate(grid_y):
        z_sample = np.array([[xi, yi]]) * 1.
        x_decoded = decoder.predict(z_sample)

        digit = x_decoded[0].reshape(digit_size, digit_size)
        figure[i * digit_size: (i + 1) * digit_size,
               j * digit_size: (j + 1) * digit_size] = digit

plt.figure(figsize=(10, 10))
plt.imshow(figure)
plt.show()

n = 15 # figure with 15x15 digits

digit_size = 28

figure = np.zeros((digit_size * n, digit_size * n))

grid_x = np.linspace(-1, 1, n)

grid_y = np.linspace(-1, 1, n)

for i, yi in enumerate(grid_x):

for j, xi in enumerate(grid_y):

z_sample = np.array([[xi, yi]]) * 1.

x_decoded = decoder.predict(z_sample)

digit = x_decoded[0].reshape(digit_size, digit_size)

figure[i * digit_size: (i + 1) * digit_size,

j * digit_size: (j + 1) * digit_size] = digit

plt.figure(figsize=(10, 10))

plt.imshow(figure)

plt.show()

Here is the output generated from sampled distribution in the above code.

The full code can be find here.

Hope you understand the basics of variational autoencoders. Hope you enjoy reading.