Category Archives: Keras

Create custom callbacks in Keras

In this blog, we will discuss how to create custom callbacks in Keras. This is actually very simple. You just need to create a class that takes keras.callbacks.Callback() as its base class. The set of methods that we can use is also fixed. We just need to write the logic. Let’s understand this with the help of an example. Here, we will create a callback that stops the training when the accuracy has reached a threshold and prints the message.

class myCallback(keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        if logs.get('acc')>0.99:
            self.model.stop_training = True
            print('Stopped training as accuracy above threshold')
    
callbacks1 = myCallback()

class myCallback(keras.callbacks.Callback):

def on_epoch_end(self, epoch, logs={}):

if logs.get('acc')>0.99:

self.model.stop_training = True

print('Stopped training as accuracy above threshold')

callbacks1 = myCallback()

In “Line-1“, we create a class “mycallback” that takes keras.callbacks.Callback() as its base class.

In “Line-2“, we define a method “on_epoch_end”. Note that the name of the functions that we can use is already predefined according to their functionality. For instance, if we define a function by the name “on_epoch_end“, then this function will be implemented at the end of every epoch. If you change this function name to “on_epoch_end1“, then this function will not be implemented.

Below are some of the method names that we can use. The name of these functions is aptly named according to their functionality. The arguments that they can take is already fixed.

The epoch and batch arguments refer to the current epoch and batch number. And “logs” is a dictionary that records all the training events like “loss”, “acc” etc.

In “Line-3,4“, we define a stopping condition and if met stop training the model. Note that we can access the model being trained through the base class. And so we can use any other model properties like save_weights, save, trainable, etc.

At last, we create an instance of this class and pass this instance as a list in the fit() method. Below is the output of applying the above callback.

Below is another example that saves the weights at the end of each epoch.

class myCallback(keras.callbacks.Callback):
    def on_batch_end(self, batch,logs={}):
        self.model.save_weights('D:/downloads/best{}.hdf5'.format(batch))
    
callbacks1 = myCallback()

class myCallback(keras.callbacks.Callback):

def on_batch_end(self, batch,logs={}):

self.model.save_weights('D:/downloads/best{}.hdf5'.format(batch))

callbacks1 = myCallback()

This way you can create many other custom callbacks in keras. This gives you more control over the training process. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – ReduceLROnPlateau

Leave a reply

As we already know that the neural networks suffer greatly from the problem of plateaus which substantially slows down the training process. Thus, the selection of a good learning rate becomes a really challenging task. Many methods have been proposed to counter this problem such as using cyclic learning rates where we vary learning rates between reasonable boundary values, or other methods like Stochastic Gradient Descent with Warm Restarts, etc.

Keras also provides ReduceLROnPlateau callback that reduces the learning rate by some factor whenever the learning stagnates. It is believed that sometimes our model will benefit from lowering the learning rate when trapped in the plateau region. So, let’s discuss its Keras API.

keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, verbose=0, mode='auto', min_delta=0.0001, cooldown=0, min_lr=0)

1	keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, verbose=0, mode='auto', min_delta=0.0001, cooldown=0, min_lr=0)

This callback “monitors” a quantity and if no improvement is seen for a ‘patience‘ number of epochs, the learning rate is reduced by the “factor” specified. Improvement is specified by the “min_delta” argument. No improvement is considered if the change in the monitored quantity is less than the min_delta specified. This also has an option whether you want to start evaluating the new LR instantly or give some time to the optimizer to crawl with the new LR and then evaluate the monitored quantity. This is done using the “cooldown” argument. You can also set the lower bound on the LR using the “min_lr” argument. No matter how many epochs or what reduction factor you use, the LR will never decrease beyond “min_lr“.

Now, let’s see how to use this.

# Load data, preprocessing and build model
...

# First, create an instance of this ReduceLROnPlateau class
from keras.callbacks import ReduceLROnPlateau
call = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=20, verbose=0, mode='auto', min_delta=0.0001, cooldown=0, min_lr=0.0001)

#Then pass this as a list in the fit() method
record = model.fit(..., callbacks=[call],...)

# Load data, preprocessing and build model

...

# First, create an instance of this ReduceLROnPlateau class

from keras.callbacks import ReduceLROnPlateau

call = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=20, verbose=0, mode='auto', min_delta=0.0001, cooldown=0, min_lr=0.0001)

#Then pass this as a list in the fit() method

record = model.fit(..., callbacks=[call],...)

That’s all for this blog. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – LambdaCallback

Leave a reply

Keras has provided several builtin classes/callbacks that serves our purpose for most of the cases. But let’s say we want to stop training when the accuracy has reached a benchmark or save the model at each batch. These tasks cannot be achieved using the builtin callbacks. In that case, we need to create our own callback function. In Keras, we can easily create custom callbacks using keras.callbacks.Callback() as our base class. But for that case, you need to create a class and write some amount of code.

As an alternative, Keras also provides us with an option to creates simple, custom callbacks on-the-fly. This can be done with ease using the LambdaCallback. So, in this blog, let’s discuss how to use this callback.

Note: For Python 3.8 or higher, lambda function will start supporting assignment expressions and this will make this callback even more powerful.

keras.callbacks.LambdaCallback(on_epoch_begin=None, on_epoch_end=None, on_batch_begin=None, on_batch_end=None, on_train_begin=None, on_train_end=None)

1	keras.callbacks.LambdaCallback(on_epoch_begin=None, on_epoch_end=None, on_batch_begin=None, on_batch_end=None, on_train_begin=None, on_train_end=None)

Here, all the arguments are aptly named according to their work. For instance, in “on_epoch_end” argument, we pass the function that will be called at the end of each epoch.

Now, all of these arguments expect functions with fixed positional arguments which are mentioned below.

Now, let’s see how to use this callback. Below I created a simple callback that saves the model weights when the accuracy is beyond some limit.

from keras.callbacks import LambdaCallback
call = LambdaCallback(on_epoch_end= lambda epochs, logs: model.save_weights('kang.hdf5') if logs.get('acc')>0.99 else None)

1 2	from keras.callbacks import LambdaCallback call = LambdaCallback(on_epoch_end= lambda epochs, logs: model.save_weights('kang.hdf5') if logs.get('acc')>0.99 else None)

Let’s take another example. Here, I’m stopping the training whenever the accuracy has reached a certain point. (For Python 3.8 or higher)

from keras.callbacks import LambdaCallback
call = LambdaCallback(on_epoch_end= lambda epochs, logs: (model.stop_training:=True) if logs.get('acc')>0.99 else None)

1 2	from keras.callbacks import LambdaCallback call = LambdaCallback(on_epoch_end= lambda epochs, logs: (model.stop_training:=True) if logs.get('acc')>0.99 else None)

Similarly, you can create any custom callback. That’s all for this blog. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – CSVLogger

Leave a reply

In this blog, we will discuss Keras CSVLogger callback. As clear from the name, this streams the training events like ‘loss’, ‘acc’ etc. to a csv file. Using this you can export all the values that can be represented as a string. So, let’s discuss its Keras API.

keras.callbacks.CSVLogger(filename, separator=',', append=False)

1	keras.callbacks.CSVLogger(filename, separator=',', append=False)

Here, the “filename” is the name of the csv file where you want to keep the record. This also gives you an option of how to separate elements in the csv file. You can pass this as a string in the “separator” argument.

This also provides an option of whether to append the training history in an existing file or overwrite the existing file. For instance, if “append=False”, this will overwrite an existing file. Otherwise, it will append the information in the existing file without affecting the previously stored information in that file.

If no existing file is present this will create a new file and then append the information. Now, let’s see how to use this class.

# Load data, preprocessing and build model
...

# First, create an instance of this CSVLogger class
from keras.callbacks import CSVLogger
call = CSVLogger('D:/train.csv', separator=',', append=False)

#Then pass this as a list in the fit() method
record = model.fit(..., callbacks=[call],...)

# Load data, preprocessing and build model

...

# First, create an instance of this CSVLogger class

from keras.callbacks import CSVLogger

call = CSVLogger('D:/train.csv', separator=',', append=False)

#Then pass this as a list in the fit() method

record = model.fit(..., callbacks=[call],...)

That’s all for this blog. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – LearningRateScheduler

Leave a reply

In neural networks, setting up a good learning rate is always a challenging task. If the learning rate is set too high, this can cause undesirable divergent behavior in your loss function or sometimes your model can converge too quickly to a sub-optimal value. If it is set too low, the training process may take a long time. Thus, it often proves sometimes useful to decay the learning rate as the training progresses. This can be done using the Learning rate schedules or the adaptive learning rate methods like SGD, Adam, etc. In this blog, we will only discuss Learning rate schedules.

Learning rate schedules as clear from the name adjusts the learning rates based on some schedule. For instance, time decay, exponential decay, etc. To implement these decays, Keras has provided a callback known as LearningRateScheduler that adjusts the weights based on the decay function provided. So, let’s discuss its Keras API.

keras.callbacks.LearningRateScheduler(schedule, verbose=0)

1	keras.callbacks.LearningRateScheduler(schedule, verbose=0)

Here, the “schedule” is a decay function that takes epoch number and the current learning rate as the input and returns the new learning rate. The verbose argument tells us whether to print the following message when changing the learning rate or not.

Note: This method overwrites the learning rate of the optimizer used.

Now, let’s discuss the schedule argument in more detail. Here, we will use the time decay function. This updates the learning rate by the expression below

Now, let’s see how to use this using the LearningRateScheduler callback. First, create a function that takes epoch and learning rate as arguments as shown below

def time_decay(epoch, initial_lrate):
    decay_rate = 0.01
    new_lrate = initial_lrate/(1+decay_rate*epoch)
    return new_lrate

def time_decay(epoch, initial_lrate):

decay_rate = 0.01

new_lrate = initial_lrate/(1+decay_rate*epoch)

return new_lrate

Then pass this function in the LearningRateScheduler callback as shown below

from keras.callbacks import LearningRateScheduler
lrate = LearningRateScheduler(time_decay,verbose=1)

1 2	from keras.callbacks import LearningRateScheduler lrate = LearningRateScheduler(time_decay,verbose=1)

Now, simply pass this callback as a list in the fit() method as shown below.

record = model.fit(..., callbacks=[lrate], ...)

1	record = model.fit(..., callbacks=[lrate], ...)

You can also check how the learning rate varies by the following command. This returns a list of learning rates over the epochs.

record.history['lr']

1	record.history['lr']

You can also plot the learning rate over epochs using any plotting library.

Similarly, you can use other decays like step decay, exponential decay or any custom decay. Just create a function and pass it to the LearningRateScheduler callback. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – ProgbarLogger

Leave a reply

In this blog, we will discuss Keras ProgbarLogger callback. As clear from the name, this deals with the logging of the progress bar that we usually see during fit() method depending upon the verbosity argument. So, let’s first discuss its API.

keras.callbacks.ProgbarLogger(count_mode='samples', stateful_metrics=None)

1	keras.callbacks.ProgbarLogger(count_mode='samples', stateful_metrics=None)

Here, “count_mode” argument controls whether the progress bar displays the samples seen or the steps. This argument can take one of the values from ‘samples‘ or ‘steps‘. If set equal to ‘steps’, make sure that you provide the “steps_per_epoch” argument in the fit() method. Otherwise, this will give you an error. The ‘steps’ is basically used with generators like fit_generator, etc.

Below is the figure where first I’ve trained on 5000 samples and the count_mode argument is set to “samples“. For the second one, I’ve used 12 steps in the fit_generator for 1 epoch. The count_mode argument is set to “steps“.

The second argument “stateful_metrics” controls whether to display the average value of the metric specified or display its value at the last step of every epoch. This should be passed as an iterable like list etc. For more details, refer to Keras callbacks BaseLogger where we have discussed this argument in detail.

This callback, in turn, calls the Keras Progbar class that controls how to display the progress bar like the width of the progress bar, its update interval, etc. Now, let’s see how to use this.

# Load data, preprocessing and build model
...

# First, create an instance of this ProgbarLogger class
from keras.callbacks import ProgbarLogger
call = ProgbarLogger(count_mode='samples', stateful_metrics=['acc'])

#Then pass this as a list in the fit() method
record = model.fit(..., callbacks=[call],...)

# Load data, preprocessing and build model

...

# First, create an instance of this ProgbarLogger class

from keras.callbacks import ProgbarLogger

call = ProgbarLogger(count_mode='samples', stateful_metrics=['acc'])

#Then pass this as a list in the fit() method

record = model.fit(..., callbacks=[call],...)

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – TerminateOnNaN

Leave a reply

In this blog, we will discuss Keras TerminateOnNaN callback. As clear from the name, this terminates the training when a Nan loss is encountered. Below is the Keras API for this callback.

keras.callbacks.TerminateOnNaN()

1	keras.callbacks.TerminateOnNaN()

This checks the loss at every batch end and if that loss is nan or inf, this callback stops the training. This prints out the batch number at which it stops the training. Something like this will be printed.

Below is the code, taken from Keras that shows how this works.

class TerminateOnNaN(Callback):
    def on_batch_end(self, batch, logs=None):
        logs = logs or {}
        loss = logs.get('loss')
        if loss is not None:
            if np.isnan(loss) or np.isinf(loss):
                print('Batch %d: Invalid loss, terminating training' % (batch))
                self.model.stop_training = True

class TerminateOnNaN(Callback):

def on_batch_end(self, batch, logs=None):

logs = logs or {}

loss = logs.get('loss')

if loss is not None:

if np.isnan(loss) or np.isinf(loss):

print('Batch %d: Invalid loss, terminating training' % (batch))

self.model.stop_training = True

Similarly, you can create your own custom callback that tracks some other metrics. Now, let’s see how to use this callback.

# Load data, preprocessing and build model
...

# First, create an instance of this TerminateOnNaN class
from keras.callbacks import TerminateOnNaN
call = TerminateOnNaN()

#Then pass this as a list in the fit() method
record = model.fit(..., callbacks=[call],...)

# Load data, preprocessing and build model

...

# First, create an instance of this TerminateOnNaN class

from keras.callbacks import TerminateOnNaN

call = TerminateOnNaN()

#Then pass this as a list in the fit() method

record = model.fit(..., callbacks=[call],...)

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – BaseLogger

Leave a reply

As we already know that the values of the metrics such as loss, acc, etc. that we get during the training are the averaged values over the epoch. This averaging of the values are automatically applied to every Keras model using the BaseLogger class present under the Keras callbacks. This class also provides us with the flexibility of not averaging the metrics over an epoch. So, in this blog, let’s discuss this BaseLogger class in more detail.

Keras API

keras.callbacks.BaseLogger(stateful_metrics=None)

1	keras.callbacks.BaseLogger(stateful_metrics=None)

Similar to the History callback, this callback is also automatically applied to every Keras model with the default set of arguments. Only if you want to change the arguments, you need to apply it similar to how we applied other callbacks i.e. pass in the fit() method.

Here, stateful_metrics are the name of the metrics that you don’t want to average over an epoch. All the metrics names should be passed in the form of iterable like lists etc. For instance, stateful_metrics=[‘acc’,’loss’].

The value of the stateful_metrics will be saved as-is in on_epoch_end. In other words, the value of the stateful_metric in the last batch before the epoch end will be saved as the final value. All the other remaining metrics will be averaged over the epoch ends. Let’s take a look at the image below.

Here, I trained the model for 1 epoch on the mnist data. The stateful_metrics used is ‘acc’. Clearly, the final logged accuracy (See record.history) is similar to the last batch accuracy (0.9392) and not the average accuracy obtained on the epoch end (0.9405). Hope this makes everything clear. Let’s see how to apply baselogger callback.

# Load data, preprocessing and build model
...

# First, create an instance of this BaseLogger class
from keras.callbacks import BaseLogger
call = BaseLogger(stateful_metrics= ['acc'])

#Then pass this as a list in the fit() method
record = model.fit(..., callbacks=[call],...)

# Load data, preprocessing and build model

...

# First, create an instance of this BaseLogger class

from keras.callbacks import BaseLogger

call = BaseLogger(stateful_metrics= ['acc'])

#Then pass this as a list in the fit() method

record = model.fit(..., callbacks=[call],...)

That’s all for this blog. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – History

Leave a reply

In neural networks, the best idea for debugging is to see the relationship between the cost and the number of iterations. This not only ensures that the optimizer is working properly but can also be very useful in the indication of overfitting. Moreover, we can also debug the learning rate based on this relationship. Thus, one should always keep a track on the loss and the accuracy metrics while training a neural network.

Fortunately, in Keras, we don’t need to write a single extra line of code to store all these values. Keras automatically keeps the record of all the events for each epoch. This includes loss and accuracy metrics for both training and validation sets (if used). This is done using the History callback which is automatically applied to every Keras model. This callback records all the events into a History object that gets returned by the fit() method.

How does this work?

First, at the onset of training, this creates an empty dictionary to store all the events. Then at every epoch end, all the events are appended into the dictionary. Below is the code for this taken from the Keras GitHub.

def on_train_begin(self, logs=None):
    self.epoch = []
    self.history = {}

def on_epoch_end(self, epoch, logs=None):
    logs = logs or {}
    self.epoch.append(epoch)
    for k, v in logs.items():
        self.history.setdefault(k, []).append(v)

def on_train_begin(self, logs=None):

self.epoch = []

self.history = {}

def on_epoch_end(self, epoch, logs=None):

logs = logs or {}

self.epoch.append(epoch)

for k, v in logs.items():

self.history.setdefault(k, []).append(v)

How to use this?

Since all the saved records are returned by the fit() method, we can simply store all the events in any variable. Here, I’ve used “record” as the variable name.

record = model.fit(train_x, train_y,validation_split=0.20, epochs=5, batch_size=128)

1	record = model.fit(train_x, train_y,validation_split=0.20, epochs=5, batch_size=128)

Now, using this record object, we can retrieve any information about the training process. For instance, “record.epoch” returns the list of epochs.

record.epoch
>>> [0, 1, 2, 3, 4]

1 2	record.epoch >>> [0, 1, 2, 3, 4]

“record.history” returns the dictionary containing the event names as the dictionary keys and their values at each epoch in a list.

record.history
>>> {'val_loss': [0.037735904552973806,
  0.0383092053757185,
  0.03597573842055863,
  0.04122187125845812,
  0.04817074624549908],
 'val_acc': [0.9889166665077209,
  0.9899166665077209,
  0.9912499998410543,
  0.989416666507721,
  0.988666666507721],
 'loss': [0.01413803466820779,
  0.010153079537209123,
  0.008668457060974712,
  0.008247203289516619,
  0.007816496806034896],
 'acc': [0.9952291666666667,
  0.9967083333333333,
  0.9972291666666667,
  0.9971875,
  0.9975208333333333]}

record.history

>>> {'val_loss': [0.037735904552973806,

0.0383092053757185,

0.03597573842055863,

0.04122187125845812,

0.04817074624549908],

'val_acc': [0.9889166665077209,

0.9899166665077209,

0.9912499998410543,

0.989416666507721,

0.988666666507721],

'loss': [0.01413803466820779,

0.010153079537209123,

0.008668457060974712,

0.008247203289516619,

0.007816496806034896],

'acc': [0.9952291666666667,

0.9967083333333333,

0.9972291666666667,

0.9971875,

0.9975208333333333]}

You can retrieve all the event names using the following command.

record.history.keys()
>>> dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])

1 2	record.history.keys() >>> dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])

You can also get the information about the parameters used while fitting the model. This can be done using the following command.

record.params
>>> {'batch_size': 128,
 'epochs': 5,
 'steps': None,
 'samples': 48000,
 'verbose': 1,
 'do_validation': True,
 'metrics': ['loss', 'acc', 'val_loss', 'val_acc']}

record.params

>>> {'batch_size': 128,

'epochs': 5,

'steps': None,

'samples': 48000,

'verbose': 1,

'do_validation': True,

'metrics': ['loss', 'acc', 'val_loss', 'val_acc']}

Not only this, but one can also check which data is used as the validation data using the following command.

record.validation_data[0]

1	record.validation_data[0]

These are just a few of functionalities available under the History callback. You can check more of these at Keras GitHub.

Plot the training history

Since all the events are stored in a dictionary, one can easily plot these using any plotting library. Here, I’m using Matplotlib. Below is the code for plotting the loss curves for both training and validation sets.

history_dict = record.history
train_acc = history_dict['loss']
val_acc = history_dict['val_loss']
epochs = range(1, len(history_dict['loss'])+1)
plt.plot(epochs, train_acc,'bo',label='Training Accuracy')
plt.plot(epochs, val_acc,'b',label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

history_dict = record.history

train_acc = history_dict['loss']

val_acc = history_dict['val_loss']

epochs = range(1, len(history_dict['loss'])+1)

plt.plot(epochs, train_acc,'bo',label='Training Accuracy')

plt.plot(epochs, val_acc,'b',label='Validation Accuracy')

plt.title('Training and Validation Accuracy')

plt.xlabel('Epochs')

plt.ylabel('Accuracy')

plt.legend()

plt.show()

Similarly, one can plot the accuracy plots. That’s all for History callback. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Keras Callbacks – EarlyStopping

Leave a reply

One common problem that we face while training a neural network is of overfitting. This refers to a situation where the model fails to generalize. In other words, the model performs poorly on the test/validation set as compared to the training set. Take a look at the plot below.

Clearly, after ‘t’ epochs, the model starts overfitting. This is clear by the increasing gap between the train and the validation error in the above plot. Wouldn’t it be nice if we stop the training where the gap starts increasing? This will help prevent the model from overfitting. This method is known as Early Stopping. Some of the pros of using this method are

Prevents the model from overfitting
Parameter-free unlike other regularization techniques like L2 etc.
Removes the need to manually set the number of epochs. Because now the model will automatically stop training when the monitored quantity stops improving.

Fortunately, in Keras, this is done using the EarlyStopping callback. So, let’s first discuss its Keras API and then we will learn how to use this.

Keras API

keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=0, verbose=0, mode='auto', baseline=None, restore_best_weights=False)

1	keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=0, verbose=0, mode='auto', baseline=None, restore_best_weights=False)

In this, you first need to provide which quantity to monitor using the “monitor” argument. This can take a value from ‘loss’, ‘acc’, ‘val_loss’, ‘val_acc’ or ‘val_metric’ where metric is the name of the metric used. For instance, if the metric is set to ‘mse’ then pass ‘val_mse’.

After setting the monitored quantity, you need to decide whether you want to minimize or maximize it. For instance, we want to minimize loss and maximize accuracy. This can be done using the “mode” argument. This can take value from [‘min‘, ‘max‘, ‘auto‘]. Default is the ‘auto’ mode. In ‘auto’ mode, this automatically infers whether to maximize or minimize depending upon the monitored quantity name.

This stops training whenever the monitored quantity stops improving. By default, any fractional change is considered as an improvement. For instance, if ‘val_acc’ increases from 90% to 90.0001% this is also considered as an improvement. The meaning of improvement may vary from one application to another. So, here we have an argument “min_delta“. Using this we can set the minimum change in the monitored quantity to qualify as an improvement. For instance, if min_delta=1, so all the absolute changes of less than 1, will count as no improvement.

Note: This difference is calculated as the current monitored quantity value minus the best-monitored quantity value until now.

As we already know that neural networks mostly face the problem of plateaus. So monitored quantity may not show improvement for some time and then improve afterward. So, it’s better to wait for a few epochs before making the final decision to stop the training process. This can be done using the “patience” argument. For instance, a patience=3 means if the monitored quantity doesn’t improve for 3 epochs, stop the training process.

The model will stop training some epochs (specified by the “patience” argument) after the best-monitored quantity value. So, the weights you will get are not the best weights. To retrieve the best weights, set the “restore_best_weights” argument to True.

Sometimes for a task, we have a baseline in our mind that at least I should get a minimum of 75% accuracy within 5 epochs. If you are not getting this, there is no point training the model any further. Then you should try changing the hyperparameters and again retrain the model. In this, you can set the baseline using the “baseline” argument. If the monitored quantity minus the min_delta is not surpassing the baseline within the epochs specified by the patience argument, then the training process is stopped.

For instance, below is an example where the baseline is set to 98%.

call = EarlyStopping(monitor='val_acc',verbose=1,min_delta=0.001,patience=3,baseline=0.99)

1	call = EarlyStopping(monitor='val_acc',verbose=1,min_delta=0.001,patience=3,baseline=0.99)

The training process stops because of the val_acc – min_delta < baseline for the patience interval (3 epochs). This is shown below.

After surpassing the baseline, the Early Stopping callback will work as normal i.e. stop training when the monitored quantity stops improving.

Note: If you are not sure about the baseline in your task, just set this argument to None.

I hope you get some feeling about the EarlyStopping callback. Now let’s see how to use this.

How to use this?

Firstly, you need to create an instance of the “EarlyStopping” class as shown below.

from keras.callbacks import EarlyStopping
earlystopping_callback = EarlyStopping(monitor='val_acc',verbose=1,min_delta=0.5,patience=3,baseline=None)

1 2	from keras.callbacks import EarlyStopping earlystopping_callback = EarlyStopping(monitor='val_acc',verbose=1,min_delta=0.5,patience=3,baseline=None)

Then pass this instance in the list while fitting the model.

record = model.fit(train_x, train_y,validation_split=0.20, epochs=10, batch_size=128,callbacks=[earlystopping_callback])

1	record = model.fit(train_x, train_y,validation_split=0.20, epochs=10, batch_size=128,callbacks=[earlystopping_callback])

That’s all for Early Stopping. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

TheAILearner

Mastering Artificial Intelligence

Category Archives: Keras

Create custom callbacks in Keras

Keras Callbacks – ReduceLROnPlateau

Keras Callbacks – LambdaCallback

Keras Callbacks – CSVLogger

Keras Callbacks – LearningRateScheduler

Keras Callbacks – ProgbarLogger

Keras Callbacks – TerminateOnNaN

Keras Callbacks – BaseLogger

Keras API

Keras Callbacks – History

How does this work?

How to use this?

Plot the training history

Keras Callbacks – EarlyStopping

Keras API

How to use this?