In the last blog we have seen autoencoders and its applications. In this blog we will learn one of its variant, sparse autoencoders.
In every autoencoder, we try to learn compressed representation of the input. Let’s take an example of a simple autoencoder having input vector dimension of 1000, compressed into 500 hidden units and reconstructed back into 1000 outputs. The hidden units will learn correlated features present in the input. But what if input features are completely random? Then it will we difficult for hidden units to learn interesting structure present in data. In that situation what we can do is increase the number of hidden units and add some sparsity constraints. Now the question is what are sparsity constraints?
When sparsity constraints added to a hidden unit, it only activates some units (having large activation values) and makes rest to zero. So, even if we are having a large number of hidden units( as in the above example), it will only fire some hidden units and learn useful structure present in the data.
The simplest implementation of sparsity constraints can be done in keras. You can
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
from keras import regularizers # creating autoencoder model encoder_inputs = Input(shape = (28,28,1)) conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(encoder_inputs) pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1) conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1) pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2) flat = Flatten()(pool2) enocder_outputs = Dense(32, activation = 'relu', activity_regularizer=regularizers.l1(10e-5))(flat) #upsampling in decoder dense_layer_d = Dense(7*7*32, activation = 'relu')(enocder_outputs) output_from_d = Reshape((7,7,32))(dense_layer_d) conv1_1 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(output_from_d) upsampling_1 = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(conv1_1) upsampling_2 = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(upsampling_1) decoded_outputs = Conv2DTranspose(1, 3, padding='same', activation='relu')(upsampling_2) autoencoder = Model(encoder_inputs, decoded_outputs) |
But, if you want to add sparse constraints by writing your own function, you can follow reference given below.
References: Sparse Autoencoders
Hope you enjoy reading.
If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.