Category Archives: ML-DL Libraries

Feeding output of a given intermediate layer in Keras as the input to another network

Keras is a high level neural network library used for fast experimentation, user friendliness and easy extensibility. It is highly recommended library for a beginner in neural networks. In this blog we will learn how to use an intermediate layer of a neural network as input to another network.

Sometimes you might get stuck while using an output of an intermediate layer with the errors like ‘graph disconnected‘. Lets see how we can solve this through the code.

First, Lets create an autoencoder model. If you are not aware of what is an autoencoder, you can follow this blog.

# creating autoencoder model
encoder_inputs = Input(shape = (28,28,1))

conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(encoder_inputs)
pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1)
conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1)
pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2)
flat = Flatten()(pool2)
encoder_outputs = Dense(32, activation = 'relu')(flat)

dense_layer_d = Dense(7*7*32, activation = 'relu')(encoder_outputs)
output_from_d = Reshape((7,7,32))(dense_layer_d)
conv1_1 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(output_from_d)
upsampling_1 = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(conv1_1)
upsampling_2 = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(upsampling_1)
decoded_outputs = Conv2DTranspose(1, 3, padding='same', activation='relu')(upsampling_2)

autoencoder = Model(encoder_inputs, decoded_outputs)

# creating autoencoder model

encoder_inputs = Input(shape = (28,28,1))

conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(encoder_inputs)

pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1)

conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1)

pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2)

flat = Flatten()(pool2)

encoder_outputs = Dense(32, activation = 'relu')(flat)

dense_layer_d = Dense(7*7*32, activation = 'relu')(encoder_outputs)

output_from_d = Reshape((7,7,32))(dense_layer_d)

conv1_1 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(output_from_d)

upsampling_1 = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(conv1_1)

upsampling_2 = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(upsampling_1)

decoded_outputs = Conv2DTranspose(1, 3, padding='same', activation='relu')(upsampling_2)

autoencoder = Model(encoder_inputs, decoded_outputs)

In the above code we have created an autoencoder model. At line 9, we have generated encoder outputs. Now if you want to create decoder network from this model with encoder_outputs layer as it input, what should you do? A beginner will do something like this:

decoder_network = Model(dense_layer_d, decoded_outputs)

1	decoder_network = Model(dense_layer_d, decoded_outputs)

But this will throw an error ‘graph disconnected’. This is because dense_layer_d layer is connected to another previous layer and you have disconnected it to directly take this layer as input. To solve this problem you can do something like this:

decoder_input = Input(shape = (32,))
next_layer = decoder_input
for layer in autoencoder.layers[-6:]:
    next_layer = layer(next_layer)

decoder = Model(decoder_input, next_layer)

decoder_input = Input(shape = (32,))

next_layer = decoder_input

for layer in autoencoder.layers[-6:]:

next_layer = layer(next_layer)

decoder = Model(decoder_input, next_layer)

Earlier we have created a model autoencoder. Now if you want to get its intermediate layer, use following steps:

Find index of the input layer to decoder( in the given autoencoder model it is the 6th layer from last so -6)
Use autoencoder.layers to get that layer.
Iterate through the following layers in the autoencoder model, till the decoder_output layer.
Then create model using decoder_input and last iterated layer.

This will successfully create a decoder model which will take the output of an intermediate layer ‘encoder_outputs’ as its input. And that’s it!!

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Custom Layers in Keras

Lambda Layer

Lambda layer is useful whenever you need to do some operation on previous layer and do not want to add any trainable weights to it.

Let say you want to add your own activation function (which is not built-in Keras) to a layer. Then you first need to define a function which will take the output from the previous layer as input and apply custom activation function to it. We then pass this function to lambda layer.

from keras.layers import Lambda
from keras import backend as K

# defining a custom non linear function
def activation_relu(inputs):
    return K.maximum(0.,inputs)

# call function using lambda layer
squashed_output = Lambda(activation_relu)(inputs) # where inputs are output from previous layer

from keras.layers import Lambda

from keras import backend as K

# defining a custom non linear function

def activation_relu(inputs):

return K.maximum(0.,inputs)

# call function using lambda layer

squashed_output = Lambda(activation_relu)(inputs) # where inputs are output from previous layer

Custom Class Layer

Sometimes you want to create your own layer with trainable weights which is not in-built in Keras. In that case you need to create a custom class layer where you need to define following methods.

__init__ method to initialize class variable and super class variables
build method to define weights.
call method where you will perform all your operations.
compute_output_shape method to define output shape of this custom layer

Lets see an example of a custom layer class. Here you only need to focus on the architecture of the class.


     class MyLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim
        super(MyLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        # Create a trainable weight variable for this layer.
        self.W= self.add_weight(name='kernel', 
                                      shape=(input_shape[1], self.output_dim),
                                      initializer='uniform',
                                      trainable=True)
        self.built = True 

    def call(self, x):
        return K.dot(x, self.W)

    def compute_output_shape(self, input_shape):
        return (input_shape[0], self.output_dim)

class MyLayer(Layer):

def __init__(self, output_dim, **kwargs):

self.output_dim = output_dim

super(MyLayer, self).__init__(**kwargs)

def build(self, input_shape):

# Create a trainable weight variable for this layer.

self.W= self.add_weight(name='kernel',

shape=(input_shape[1], self.output_dim),

initializer='uniform',

trainable=True)

self.built = True

def call(self, x):

return K.dot(x, self.W)

def compute_output_shape(self, input_shape):

return (input_shape[0], self.output_dim)

In the build method defining self.built = True is necessary. Also, you can see that all logic is written inside call(self, inputs) method. comput_output_shape will define the output shape of the layer.

You can also pass multiple input tensor to this custom layer. The only thing you need to do is, pass multiple inputs using a list.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Saving and Loading models in Keras

1 Reply

Generally, a deep learning model takes a large amount of time to train, so its better to know how to save trained model. In this blog we will learn about how to save whole keras model i.e. its architecture, weights and optimizer state.

Lets first create a model in Keras. This is a simple autoencoder model. If you need to know more about autoencoders please refer this blog.

from keras.models import Model
from keras.layers import Dense, Input
import matplotlib.pyplot as plt
import numpy as np

# creating an autoencoder model
input_shape = Input(shape = (784,))
dense1 = Dense(512, activation = 'relu')(input_shape)
dense2 = Dense(256, activation = 'relu')(dense1)
encoded = Dense(32, activation = 'relu')(dense2)

dense3 = Dense(256, activation = 'relu')(encoded)
dense4 = Dense(512, activation = 'relu')(dense3)
decoded = Dense(784, activation = 'relu')(dense4)

autoencoder = Model(input_shape, decoded)

m = 256
n_epoch = 25
autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.fit(output_X_train,output_X_train, epochs=n_epoch, batch_size=m, shuffle=True)

from keras.models import Model

from keras.layers import Dense, Input

import matplotlib.pyplot as plt

import numpy as np

# creating an autoencoder model

input_shape = Input(shape = (784,))

dense1 = Dense(512, activation = 'relu')(input_shape)

dense2 = Dense(256, activation = 'relu')(dense1)

encoded = Dense(32, activation = 'relu')(dense2)

dense3 = Dense(256, activation = 'relu')(encoded)

dense4 = Dense(512, activation = 'relu')(dense3)

decoded = Dense(784, activation = 'relu')(dense4)

autoencoder = Model(input_shape, decoded)

m = 256

n_epoch = 25

autoencoder.compile(optimizer='adam', loss='mse')

autoencoder.fit(output_X_train,output_X_train, epochs=n_epoch, batch_size=m, shuffle=True)

Above we have created a Keras model named as “autoencoder“. Now lets see how to save this model.

Saving and loading only architecture of a model

In keras, you can save and load architecture of a model in two formats: JSON or YAML Models generated in these two format are human readable and can be edited if needed.

# saving and loading model architecture in json format

# saving in json format
json_model = autoencoder.to_json()
json_file = open('autoencoder_json.json', 'w')
json_file.write(json_model)

# loading model architecture from json file
from keras.models import model_from_json
json_file = open('autoencoder_json.json', 'r')
json_model = model_from_json(json_file.read())

# saving and loading model architecture in json format

# saving in json format

json_model = autoencoder.to_json()

json_file = open('autoencoder_json.json', 'w')

json_file.write(json_model)

# loading model architecture from json file

from keras.models import model_from_json

json_file = open('autoencoder_json.json', 'r')

json_model = model_from_json(json_file.read())

# saving and loading model architecture in yaml format

# saving in yaml format
yaml_model = autoencoder.to_yaml()
yaml_file = open('autoencoder_yaml.yaml', 'w')
yaml_file.write(yaml_model)

# loading model architecture from yaml file
from keras.models import model_from_yaml
yaml_file = open('autoencoder_yaml.yaml', 'r')
yaml_model = model_from_yaml(yaml_file.read())

# saving and loading model architecture in yaml format

# saving in yaml format

yaml_model = autoencoder.to_yaml()

yaml_file = open('autoencoder_yaml.yaml', 'w')

yaml_file.write(yaml_model)

# loading model architecture from yaml file

from keras.models import model_from_yaml

yaml_file = open('autoencoder_yaml.yaml', 'r')

yaml_model = model_from_yaml(yaml_file.read())

Saving and Loading Weights of a Keras Model

With model architecture you will also need model weights to predict output from trained model.

# saving model weights
autoencoder.save_weights('autoencoder_weights.h5')

# loading weights of a keras model
json_model.load_weights('autoencoder_weights.h5')

# saving model weights

autoencoder.save_weights('autoencoder_weights.h5')

# loading weights of a keras model

json_model.load_weights('autoencoder_weights.h5')

Saving and Loading Both Architecture and Weights in one File

# saving whole model
autoencoder.save('autoencoder_model.h5')

# loading whole model
from keras.models import load_model
model1 = load_model('autoencoder_model.h5')

# saving whole model

autoencoder.save('autoencoder_model.h5')

# loading whole model

from keras.models import load_model

model1 = load_model('autoencoder_model.h5')

This will save following four parameters in “autoencoder_model.h5” file:

Model Architecture
Model Weights
Loss and Optimizer
State of the optimizer allowing to resume training where you left.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

TheAILearner

Mastering Artificial Intelligence

Category Archives: ML-DL Libraries

Feeding output of a given intermediate layer in Keras as the input to another network

Custom Layers in Keras

Lambda Layer

Custom Class Layer

Saving and Loading models in Keras