Tag Archives: python

Show current DateTime on live video using OpenCV-Python

Have you seen the security cameras output where DateTime continuously keeps updating? In this blog, we will be doing the same using OpenCV-Python i.e. we will put current DateTime on the live webcam feed. So, let’s get started.

For fetching current DateTime, we will be using Python’s DateTime module. The following code shows how to get the current DateTime

# get current DateTime
from datetime import datetime
print(datetime.now())

# get current DateTime

from datetime import datetime

print(datetime.now())

To put the DateTime on the live video, we will be using cv2.putText() on each frame as shown below

# Put current DateTime on each frame
cv2.putText(img,str(datetime.now()),(140,250), font, .5,(255,255,255),2,cv2.LINE_AA)

1 2	# Put current DateTime on each frame cv2.putText(img,str(datetime.now()),(140,250), font, .5,(255,255,255),2,cv2.LINE_AA)

To know more about cv2.putText(), refer to this blog.

Above are the two things, that we will be needing for this task. I hope you understand these. Now, let’s get started

Steps:

Open the camera using cv2.VideoCapture()
Until the camera is open
- Grab each frame using cap.read()
- Put the current DateTime on each frame using cv2.putText() as discussed above
- Display each frame using cv2.imshow()
On termination, release the webcam and destroy all windows using cap.release() and cv2.destroyAllWindows() respectively.

Code:

import cv2
from datetime import datetime
# Open the Camera
cap = cv2.VideoCapture(0)
while True:
    ret, img = cap.read()
    # Put current DateTime on each frame
    font = cv2.FONT_HERSHEY_SIMPLEX
    cv2.putText(img,str(datetime.now()),(10,30), font, 1,(255,255,255),2,cv2.LINE_AA)
    # Display the image
    cv2.imshow('a',img)
    # wait for keypress
    k = cv2.waitKey(10)
    if k == ord('q'):
        break
cap.release()
cv2.destroyAllWindows()

import cv2

from datetime import datetime

# Open the Camera

cap = cv2.VideoCapture(0)

while True:

ret, img = cap.read()

# Put current DateTime on each frame

font = cv2.FONT_HERSHEY_SIMPLEX

cv2.putText(img,str(datetime.now()),(10,30), font, 1,(255,255,255),2,cv2.LINE_AA)

# Display the image

cv2.imshow('a',img)

# wait for keypress

k = cv2.waitKey(10)

if k == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

The snapshot of the output looks like this

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Scraping Video Information from YouTube

3 Replies

Web scraping is a way to extract information from the internet in an automated fashion. We all know that YouTube is a huge resource of data having tons of videos with their relative information’s like views, comments, etc.In this blog we will learn how to use web scraping in python to extract video information from YouTube search. In video information we will extract number of views and video heading appeared in search results.

To get started with this, we first need to install two important libraries. First is ” requests ” to get the response from a YouTube search result and other is ” Beautiful Soup ” to parse this response into html content.

pip install requests
pip install -U bs4

1 2	pip install requests pip install -U bs4

Now we have install the required libraries, let’s get started.

Import the libraries

from bs4 import BeautifulSoup as bs
import requests

1 2	from bs4 import BeautifulSoup as bs import requests

Whenever you search in YouTube, it creates a base search URL and then adds your search query into that URL to complete the it. Let say we search ” theailearner ” in the YouTube. Base search URL and query can be defined as follows.

base_url = 'https://www.youtube.com/results?search_query='
search_string = 'theailearner'
URL = base_url + search_string

base_url = 'https://www.youtube.com/results?search_query='

search_string = 'theailearner'

URL = base_url + search_string

Now, we will scrape the data from this URL using ” requests ” library.

response = requests.get(URL)
page = response.text

1 2	response = requests.get(URL) page = response.text

Once we scraped the data, we will parse it into HTML using beautiful soup and find all the videos information resulted in search result. To extract particular information we will use particular class from HTML data.

soup = bs(page, 'html.parser')
vids = soup.findAll('a',attrs={'class':'yt-uix-tile-link'})

1 2	soup = bs(page, 'html.parser') vids = soup.findAll('a',attrs={'class':'yt-uix-tile-link'})

The above used soup.findall() function will give the required data, but to make it easily understandable we need to run a simple python script.

for v in vids:
    print(v['title'])
    v = str(v)
    views = ''
    try:
        indx = v.index('views')
        indx = indx - 2
        while v[indx] is not ' ':
            views = views + v[indx]
            indx = indx -1
        print(views[::-1])
    except:
        continue

for v in vids:

print(v['title'])

v = str(v)

views = ''

try:

indx = v.index('views')

indx = indx - 2

while v[indx] is not ' ':

views = views + v[indx]

indx = indx -1

print(views[::-1])

except:

continue

Now you might have got some feeling about how to scrape data from YouTube. We can also scrape the other data from YouTube like video information from a channel, comments in a video, likes and dislikes and etc.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Creating a Snake Game using OpenCV-Python

Import Libraries

For this we only need four libraries

import numpy as np
import cv2
import random
import time

import numpy as np

import cv2

import random

import time

Displaying Game Objects

Game Window: Here, I have used a 500×500 image as my game window.

img = np.zeros((500,500,3),dtype='uint8')

1	img = np.zeros((500,500,3),dtype='uint8')

Snake and Apple: I have used green squares for displaying a snake and a red square for an apple. Each square has a size of 10 units.

# Displaying the snake (Green rectangles)
snake_position = [[250,250],[240,250],[230,250]]
for position in snake_position:
        cv2.rectangle(img,(position[0],position[1]),(position[0]+10,position[1]+10),(0,255,0),3)

# Displaying the snake (Green rectangles)

snake_position = [[250,250],[240,250],[230,250]]

for position in snake_position:

cv2.rectangle(img,(position[0],position[1]),(position[0]+10,position[1]+10),(0,255,0),3)

# Display apple (Red rectangles)
apple_position = [random.randrange(1,50)*10,random.randrange(1,50)*10]
cv2.rectangle(img,(apple_position[0],apple_position[1]),(apple_position[0]+10,apple_position[1]+10),(0,0,255),3)

# Display apple (Red rectangles)

apple_position = [random.randrange(1,50)*10,random.randrange(1,50)*10]

cv2.rectangle(img,(apple_position[0],apple_position[1]),(apple_position[0]+10,apple_position[1]+10),(0,0,255),3)

Game Rules

Now, let’s define some game rules

Collision with boundaries: If the snake collides with the boundaries, it dies.

def collision_with_boundaries(snake_head):
    if snake_head[0]>=500 or snake_head[0]<=0 or snake_head[1]>=500 or snake_head[1]<=0 :
        return 1
    else:
        return 0

def collision_with_boundaries(snake_head):

if snake_head[0]>=500 or snake_head[0]<=0 or snake_head[1]>=500 or snake_head[1]<=0 :

return 1

else:

return 0

Collision with self: If the snake collides with itself, it should die. For this, we only need to check whether the snake’s head is in snake body or not.

def collision_with_self(snake_position):
    snake_head = snake_position[0]
    if snake_head in snake_position[1:]:
        return 1
    else:
        return 0

def collision_with_self(snake_position):

snake_head = snake_position[0]

if snake_head in snake_position[1:]:

return 1

else:

return 0

Collision with apple: If the snake collides with the apple, the score is increased and the apple is moved to a new location.

def collision_with_apple(apple_position, score):
    apple_position = [random.randrange(1,50)*10,random.randrange(1,50)*10]
    score += 1
    return apple_position, score

def collision_with_apple(apple_position, score):

apple_position = [random.randrange(1,50)*10,random.randrange(1,50)*10]

score += 1

return apple_position, score

Also, on eating apple snake length should increase. Otherwise, snake moves as it is.

if snake_head == apple_position:
        apple_position, score = collision_with_apple(apple_position, score)
        snake_position.insert(0,list(snake_head))

    else:
        snake_position.insert(0,list(snake_head))
        snake_position.pop()

if snake_head == apple_position:

apple_position, score = collision_with_apple(apple_position, score)

snake_position.insert(0,list(snake_head))

else:

snake_position.insert(0,list(snake_head))

snake_position.pop()

Snake game has a fixed time for a keypress. If you press any button in that time, the snake should move in that direction otherwise continue moving in the previous direction. Sadly, with OpenCV cv2.waitKey() function, if you hold down the left direction button, the snake starts moving fast in that direction. So, to make the snake movement uniform, i did something like this.

    t_end = time.time() + 0.2
    k = -1
    while time.time() < t_end:
        if k == -1:
            k = cv2.waitKey(125)
        else:
            continue

t_end = time.time() + 0.2

k = -1

while time.time() < t_end:

if k == -1:

k = cv2.waitKey(125)

else:

continue

Because cv2.waitKey() returns -1 when no key is pressed, so this ‘k’ stores the first key pressed in that time. Because the while loop is for a fixed time, so it doesn’t matter how fast you pressed a key. It will always wait a fixed time.

Snake cannot move backward: Here, I have used the w, a, s, d controls for moving the snake. If the snake was moving right and we pressed the left button, it will continue moving right or in short snake cannot directly move backwards.

# 0-Left, 1-Right, 3-Up, 2-Down, q-Break
# a-Left, d-Right, w-Up, s-Down
if k == ord('a') and prev_button_direction != 1:
    button_direction = 0
elif k == ord('d') and prev_button_direction != 0:
    button_direction = 1
elif k == ord('w') and prev_button_direction != 2:
    button_direction = 3
elif k == ord('s') and prev_button_direction != 3:
    button_direction = 2
elif k == ord('q'):
    break
else:
    button_direction = button_direction

# 0-Left, 1-Right, 3-Up, 2-Down, q-Break

# a-Left, d-Right, w-Up, s-Down

if k == ord('a') and prev_button_direction != 1:

button_direction = 0

elif k == ord('d') and prev_button_direction != 0:

button_direction = 1

elif k == ord('w') and prev_button_direction != 2:

button_direction = 3

elif k == ord('s') and prev_button_direction != 3:

button_direction = 2

elif k == ord('q'):

break

else:

button_direction = button_direction

After seeing which direction button is pressed, we change our head position

if button_direction == 1:
    snake_head[0] += 10
elif button_direction == 0:
    snake_head[0] -= 10
elif button_direction == 2:
    snake_head[1] += 10
elif button_direction == 3:
    snake_head[1] -= 10

if button_direction == 1:

snake_head[0] += 10

elif button_direction == 0:

snake_head[0] -= 10

elif button_direction == 2:

snake_head[1] += 10

elif button_direction == 3:

snake_head[1] -= 10

Displaying the final Score

For displaying the final score, i have used cv2.putText() function.

if collision_with_boundaries(snake_head) == 1 or collision_with_self(snake_position) == 1:
    font = cv2.FONT_HERSHEY_SIMPLEX
    img = np.zeros((500,500,3),dtype='uint8')
    cv2.putText(img,'Your Score is {}'.format(score),(140,250), font, 1,(255,255,255),2,cv2.LINE_AA)
    cv2.imshow('a',img)
    cv2.waitKey(0)
    break

if collision_with_boundaries(snake_head) == 1 or collision_with_self(snake_position) == 1:

font = cv2.FONT_HERSHEY_SIMPLEX

img = np.zeros((500,500,3),dtype='uint8')

cv2.putText(img,'Your Score is {}'.format(score),(140,250), font, 1,(255,255,255),2,cv2.LINE_AA)

cv2.imshow('a',img)

cv2.waitKey(0)

break

Finally, our snake game is ready and looks like this

The full code can be found here.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Snake Game Using Tensorflow Object Detection API – Part IV

1 Reply

In the last blog we have trained the model and saved the inference graph. In this blog we will learn how to use this inference graph for object detection and how to run our snake game using this trained object detection model.

To play snake game using this trained model, you first need to develop a snake game. But don’t worry you need not to develop it from scratch, you can clone this repository. And if you want to know algorithm behind this code you can follow this blog.

Now we have our snake game next thing is to use this object detection model to play the snake game. To do this we need to run both snake game file and following script from models/research folder simultaneously.

import os
import sys
from multiprocessing import Value

import cv2
import numpy as np
import pyautogui
import tensorflow as tf

cap = cv2.VideoCapture(0)

sys.path.append("..")

from object_detection.utils import label_map_util

from object_detection.utils import visualization_utils as vis_util

# # Model preparation

# Path to frozen detection graph. This is the actual model that is used for the object detection.
PATH_TO_CKPT = 'snake/frozen_inference_graph.pb'

# List of the strings that is used to add correct label for each box.
PATH_TO_LABELS = os.path.join('images/data', 'object-detection.pbtxt')

NUM_CLASSES = 4

# ## Load a (frozen) Tensorflow model into memory.
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

# ## Loading label map
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES,
                                                            use_display_name=True)
category_index = label_map_util.create_category_index(categories)


with detection_graph.as_default():
#     from directkeys import PressKey, ReleaseKey, W

    # enter your monitor's resolution or use a library to fetch this - I had to hard code due to issues with
    # dual monitor setup
    x, y = 288, 512

    # init process safe variables for workers
    objectX, objectY = Value('d', 0.0), Value('d', 0.0)
    objectX_previous = None
    objectY_previous = None
    with tf.Session(graph=detection_graph) as sess:
        # Definite input and output Tensors for detection_graph
        image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
        # Each box represents a part of the image where a particular object was detected.
        detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
        # Each score represent how level of confidence for each of the objects.
        # Score is shown on the result image, together with the class label.
        detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
        detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')
        num_detections = detection_graph.get_tensor_by_name('num_detections:0')
        while True:
            ret, image_np = cap.read()
            # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
            image_np_expanded = np.expand_dims(image_np, axis=0)
            # Actual detection.
            (boxes, scores, classes, num) = sess.run(
                [detection_boxes, detection_scores, detection_classes, num_detections],
                feed_dict={image_tensor: image_np_expanded})
            # Visualization of the results of a detection.
            vis_util.visualize_boxes_and_labels_on_image_array(
                image_np,
                np.squeeze(boxes),
                np.squeeze(classes).astype(np.int32),
                np.squeeze(scores),
                category_index,
                use_normalized_coordinates=True,
                line_thickness=8)
            cv2.imshow('controls detection', image_np)
            if cv2.waitKey(50) & 0xFF == ord('q'):
                cv2.destroyAllWindows()
                break


            '''MOVE'''
            # press 'w' if bounding box of finger detected
            objects = np.where(classes[0] == 1)[0]

            # calculate center of box if detection exceeds threshold
            if len(objects) > 0 and scores[0][objects][0] > 0.15:
                pyautogui.press('up')

                
            objects = np.where(classes[0] == 2)[0]

            # calculate center of box if detection exceeds threshold
            if len(objects) > 0 and scores[0][objects][0] > 0.15:
                pyautogui.press('down')
                
                
            objects = np.where(classes[0] == 3)[0]

            # calculate center of box if detection exceeds threshold
            if len(objects) > 0 and scores[0][objects][0] > 0.15:
                pyautogui.press('left')
                
                
            objects = np.where(classes[0] == 4)[0]

            # calculate center of box if detection exceeds threshold
            if len(objects) > 0 and scores[0][objects][0] > 0.15:
                pyautogui.press('right')
                
            
cap.release()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

import os

import sys

from multiprocessing import Value

import cv2

import numpy as np

import pyautogui

import tensorflow as tf

cap = cv2.VideoCapture(0)

sys.path.append("..")

from object_detection.utils import label_map_util

from object_detection.utils import visualization_utils as vis_util

# # Model preparation

# Path to frozen detection graph. This is the actual model that is used for the object detection.

PATH_TO_CKPT = 'snake/frozen_inference_graph.pb'

# List of the strings that is used to add correct label for each box.

PATH_TO_LABELS = os.path.join('images/data', 'object-detection.pbtxt')

NUM_CLASSES = 4

# ## Load a (frozen) Tensorflow model into memory.

detection_graph = tf.Graph()

with detection_graph.as_default():

od_graph_def = tf.GraphDef()

with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:

serialized_graph = fid.read()

od_graph_def.ParseFromString(serialized_graph)

tf.import_graph_def(od_graph_def, name='')

# ## Loading label map

label_map = label_map_util.load_labelmap(PATH_TO_LABELS)

categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES,

use_display_name=True)

category_index = label_map_util.create_category_index(categories)

with detection_graph.as_default():

# from directkeys import PressKey, ReleaseKey, W

# enter your monitor's resolution or use a library to fetch this - I had to hard code due to issues with

# dual monitor setup

x, y = 288, 512

# init process safe variables for workers

objectX, objectY = Value('d', 0.0), Value('d', 0.0)

objectX_previous = None

objectY_previous = None

with tf.Session(graph=detection_graph) as sess:

# Definite input and output Tensors for detection_graph

image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')

# Each box represents a part of the image where a particular object was detected.

detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')

# Each score represent how level of confidence for each of the objects.

# Score is shown on the result image, together with the class label.

detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')

detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')

num_detections = detection_graph.get_tensor_by_name('num_detections:0')

while True:

ret, image_np = cap.read()

# Expand dimensions since the model expects images to have shape: [1, None, None, 3]

image_np_expanded = np.expand_dims(image_np, axis=0)

# Actual detection.

(boxes, scores, classes, num) = sess.run(

[detection_boxes, detection_scores, detection_classes, num_detections],

feed_dict={image_tensor: image_np_expanded})

# Visualization of the results of a detection.

vis_util.visualize_boxes_and_labels_on_image_array(

image_np,

np.squeeze(boxes),

np.squeeze(classes).astype(np.int32),

np.squeeze(scores),

category_index,

use_normalized_coordinates=True,

line_thickness=8)

cv2.imshow('controls detection', image_np)

if cv2.waitKey(50) & 0xFF == ord('q'):

cv2.destroyAllWindows()

break

'''MOVE'''

# press 'w' if bounding box of finger detected

objects = np.where(classes[0] == 1)[0]

# calculate center of box if detection exceeds threshold

if len(objects) > 0 and scores[0][objects][0] > 0.15:

pyautogui.press('up')

objects = np.where(classes[0] == 2)[0]

# calculate center of box if detection exceeds threshold

if len(objects) > 0 and scores[0][objects][0] > 0.15:

pyautogui.press('down')

objects = np.where(classes[0] == 3)[0]

# calculate center of box if detection exceeds threshold

if len(objects) > 0 and scores[0][objects][0] > 0.15:

pyautogui.press('left')

objects = np.where(classes[0] == 4)[0]

# calculate center of box if detection exceeds threshold

if len(objects) > 0 and scores[0][objects][0] > 0.15:

pyautogui.press('right')

cap.release()

In the above code we need to specify path to our inference graph using ” PATH_TO_CKPT ” variable. Also we need to specify ” PATH_TO_LABELS ” variable with path of object-detection.pbtxt file. Then specify number of classes i.e. 4 in our case.

In the above script we have used ” pyautogui ” to press the button when particular hand gesture for a particular direction is detected.

Finally you can play snake game using your hand gestures. Let see some of the results.

Pretty well yeah. This is all for playing snake game using tensorflow object detection API. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Snake Game Using Tensorflow Object Detection API – Part III

1 Reply

In the previous blogs we have seen how to generate data for object detection and convert it into TFRecord format to train the model. In this blog we will learn how to use this data to train the model.

To train the model we will use the pre-trained model and then use transfer learning to train it on our dataset. I have used mobilenet pre trained model. Here is mobilenet model. For its configuration file you can go to model -> research -> object_detection -> samples -> configs ->> ssd_mobilenet_v1_pets.config.
The configuration file that we have downloaded, needs to be edited as per our requirement. In configuration file we have changed the no. of classes, no. of steps in training, path to model checkpoint and path to pbtxt files as shown below.

# SSD with Mobilenet v1, configured for Oxford-IIIT Pets Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.

model {
  ssd {
    num_classes: 4
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    anchor_generator {
      ssd_anchor_generator {
        num_layers: 6
        min_scale: 0.2
        max_scale: 0.95
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 3.0
        aspect_ratios: 0.3333
      }
    }
    image_resizer {
      fixed_shape_resizer {
        height: 300
        width: 300
      }
    }
    box_predictor {
      convolutional_box_predictor {
        min_depth: 0
        max_depth: 0
        num_layers_before_predictor: 0
        use_dropout: false
        dropout_keep_probability: 0.8
        kernel_size: 1
        box_code_size: 4
        apply_sigmoid_to_scores: false
        conv_hyperparams {
          activation: RELU_6,
          regularizer {
            l2_regularizer {
              weight: 0.00004
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.03
              mean: 0.0
            }
          }
          batch_norm {
            train: true,
            scale: true,
            center: true,
            decay: 0.9997,
            epsilon: 0.001,
          }
        }
      }
    }
    feature_extractor {
      type: 'ssd_mobilenet_v1'
      min_depth: 16
      depth_multiplier: 1.0
      conv_hyperparams {
        activation: RELU_6,
        regularizer {
          l2_regularizer {
            weight: 0.00004
          }
        }
        initializer {
          truncated_normal_initializer {
            stddev: 0.03
            mean: 0.0
          }
        }
        batch_norm {
          train: true,
          scale: true,
          center: true,
          decay: 0.9997,
          epsilon: 0.001,
        }
      }
    }
    loss {
      classification_loss {
        weighted_sigmoid {
          anchorwise_output: true
        }
      }
      localization_loss {
        weighted_smooth_l1 {
          anchorwise_output: true
        }
      }
      hard_example_miner {
        num_hard_examples: 3000
        iou_threshold: 0.99
        loss_type: CLASSIFICATION
        max_negatives_per_positive: 3
        min_negatives_per_image: 0
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    normalize_loss_by_num_matches: true
    post_processing {
      batch_non_max_suppression {
        score_threshold: 1e-8
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SIGMOID
    }
  }
}

train_config: {
  batch_size: 16
  optimizer {
    rms_prop_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.004
          decay_steps: 800720
          decay_factor: 0.95
        }
      }
      momentum_optimizer_value: 0.9
      decay: 0.9
      epsilon: 1.0
    }
  }
  fine_tune_checkpoint: "D:/models/research/images/ssd_mobilenet_v1_coco_11_06_2017/model.ckpt"
  from_detection_checkpoint: true
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the pets dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps: 200000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    ssd_random_crop {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "D:/models/research/images/data/train.record"
  }
  label_map_path: "D:/models/research/images/data/object-detection.pbtxt"
}

eval_config: {
  num_examples: 2000
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "D:/models/research/images/data/test.record"
  }
  label_map_path: "D:/models/research/images/data/object-detection.pbtxt"
  shuffle: false
  num_readers: 1
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

# SSD with Mobilenet v1, configured for Oxford-IIIT Pets Dataset.

# Users should configure the fine_tune_checkpoint field in the train config as

# well as the label_map_path and input_path fields in the train_input_reader and

# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that

# should be configured.

model {

ssd {

num_classes: 4

box_coder {

faster_rcnn_box_coder {

y_scale: 10.0

x_scale: 10.0

height_scale: 5.0

width_scale: 5.0

}

matcher {

argmax_matcher {

matched_threshold: 0.5

unmatched_threshold: 0.5

ignore_thresholds: false

negatives_lower_than_unmatched: true

force_match_for_each_row: true

}

similarity_calculator {

iou_similarity {

}

anchor_generator {

ssd_anchor_generator {

num_layers: 6

min_scale: 0.2

max_scale: 0.95

aspect_ratios: 1.0

aspect_ratios: 2.0

aspect_ratios: 0.5

aspect_ratios: 3.0

aspect_ratios: 0.3333

}

image_resizer {

fixed_shape_resizer {

height: 300

width: 300

}

box_predictor {

convolutional_box_predictor {

min_depth: 0

max_depth: 0

num_layers_before_predictor: 0

use_dropout: false

dropout_keep_probability: 0.8

kernel_size: 1

box_code_size: 4

apply_sigmoid_to_scores: false

conv_hyperparams {

activation: RELU_6,

regularizer {

l2_regularizer {

weight: 0.00004

}

initializer {

truncated_normal_initializer {

stddev: 0.03

mean: 0.0

}

batch_norm {

train: true,

scale: true,

center: true,

decay: 0.9997,

epsilon: 0.001,

}

feature_extractor {

type: 'ssd_mobilenet_v1'

min_depth: 16

depth_multiplier: 1.0

conv_hyperparams {

activation: RELU_6,

regularizer {

l2_regularizer {

weight: 0.00004

}

initializer {

truncated_normal_initializer {

stddev: 0.03

mean: 0.0

}

batch_norm {

train: true,

scale: true,

center: true,

decay: 0.9997,

epsilon: 0.001,

}

loss {

classification_loss {

weighted_sigmoid {

anchorwise_output: true

}

localization_loss {

weighted_smooth_l1 {

anchorwise_output: true

}

hard_example_miner {

num_hard_examples: 3000

iou_threshold: 0.99

loss_type: CLASSIFICATION

max_negatives_per_positive: 3

min_negatives_per_image: 0

}

classification_weight: 1.0

localization_weight: 1.0

}

normalize_loss_by_num_matches: true

post_processing {

batch_non_max_suppression {

score_threshold: 1e-8

iou_threshold: 0.6

max_detections_per_class: 100

max_total_detections: 100

}

score_converter: SIGMOID

}

train_config: {

batch_size: 16

optimizer {

rms_prop_optimizer: {

learning_rate: {

exponential_decay_learning_rate {

initial_learning_rate: 0.004

decay_steps: 800720

decay_factor: 0.95

}

momentum_optimizer_value: 0.9

decay: 0.9

epsilon: 1.0

}

fine_tune_checkpoint: "D:/models/research/images/ssd_mobilenet_v1_coco_11_06_2017/model.ckpt"

from_detection_checkpoint: true

# Note: The below line limits the training process to 200K steps, which we

# empirically found to be sufficient enough to train the pets dataset. This

# effectively bypasses the learning rate schedule (the learning rate will

# never decay). Remove the below line to train indefinitely.

num_steps: 200000

data_augmentation_options {

random_horizontal_flip {

}

data_augmentation_options {

ssd_random_crop {

}

train_input_reader: {

tf_record_input_reader {

input_path: "D:/models/research/images/data/train.record"

}

label_map_path: "D:/models/research/images/data/object-detection.pbtxt"

}

eval_config: {

num_examples: 2000

# Note: The below line limits the evaluation process to 10 evaluations.

# Remove the below line to evaluate indefinitely.

max_evals: 10

}

eval_input_reader: {

tf_record_input_reader {

input_path: "D:/models/research/images/data/test.record"

}

label_map_path: "D:/models/research/images/data/object-detection.pbtxt"

shuffle: false

num_readers: 1

}

For the object-detection.pbtxt file, create a pbtxt file and put following text inside it to specify our labels for the problem.

item {
  id: 1
  name: 'up'
}
item {
	id:2
	name:"down"
}
item {
	id:3
	name:"left"
}
item {
	id:4
	name:"right"
}

item {

id: 1

name: 'up'

}

item {

id:2

name:"down"

}

item {

id:3

name:"left"

}

item {

id:4

name:"right"

}

Now go to models -> research -> object detection -> legecy and copy train.py file to models -> research folder.

Then create a folder named images inside models -> research folder. Put your mobilenet model, configuration file, train and test image data folders, and train and test csv label files. Inside training_data folder, create a folder named data and put your train and test TFRecord files. The hierarchy will look like this:

images
   -data
      -object-detection.pbtxt
	  -test.record
	  -train.record
   -test
      -contains all test images data
   -train
      -contains all train images data
   -ssd_mobilenet_v1_pets.config
   -test_labels.csv
   -train_labels.csv
   -training

images

-data

-object-detection.pbtxt

-test.record

-train.record

-test

-contains all test images data

-train

-contains all train images data

-ssd_mobilenet_v1_pets.config

-test_labels.csv

-train_labels.csv

-training

Also create a training folder inside the images folder where model will save its checkpoints. Now run the following command to train the model from models -> research folder.

python train.py --logtostderr --train_dir=images/training/ --pipeline_config_path=images/ssd_mobilenet_v1_pets.config

1	python train.py --logtostderr --train_dir=images/training/ --pipeline_config_path=images/ssd_mobilenet_v1_pets.config

Time for training your model will depend upon your machine configuration and no. of steps that you have mentioned in the configuration file.

Now we have our trained model and its checkpoints are saved inside the models/research/images/training folder. In order to test this model and use this model to detect objects we need to export the inference graph.

To do this first we need to copy models/research/object_detection/export_inference_graph.py to models/research/ folder. Then inside models/research folder create a folder named “snake” which will save the inference graph. From models -> research folder run the following command:

python export_inference_graph.py --input_type image_tensor --pipeline_config_path images/ssd_mobilenet_v1_pets.config --trained_checkpoint_prefix images/training/model.ckpt-34345 --output_directory snake

1	python export_inference_graph.py --input_type image_tensor --pipeline_config_path images/ssd_mobilenet_v1_pets.config --trained_checkpoint_prefix images/training/model.ckpt-34345 --output_directory snake

Now we are having forzen_inference_graph.pb inside models/research/snake folder which will be used to detect object using trained model.

This is all for training the model and saving the inference graph, in the next blog we will see how to use this inference graph for object detection and how to run our snake game using this trained object detection model.

Next Blog: Snake Game Using Tensorflow Object Detection API – Part IV

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Snake Game Using Tensorflow Object Detection API – Part II

1 Reply

In the previous blog, we did two things. First, we create a dataset and second we split this into training and test. In this blog, we will learn how to convert this dataset into TFRecord format for training.

Before creating TFRecord file, we just need to do one more step. In the last blog, we have generated XML files using LabelImg. To get labels for training and test dataset, we need to convert these XML files into CSV format. To do this we will use the following code which has been taken from this repository.

import os
import glob
import pandas as pd
import xml.etree.ElementTree as ET


def xml_to_csv(path):
    xml_list = []
    for xml_file in glob.glob(path + '/*.xml'):
        tree = ET.parse(xml_file)
        root = tree.getroot()
        for member in root.findall('object'):
            value = (root.find('filename').text,
                     int(root.find('size')[0].text),
                     int(root.find('size')[1].text),
                     member[0].text,
                     int(member[4][0].text),
                     int(member[4][1].text),
                     int(member[4][2].text),
                     int(member[4][3].text)
                     )
            xml_list.append(value)
    column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']
    xml_df = pd.DataFrame(xml_list, columns=column_name)
    return xml_df


def main():
    for directory in ['train','test']:
        image_path = os.path.join(os.getcwd(), 'images/{}'.format(directory))
        xml_df = xml_to_csv(image_path)
        xml_df.to_csv('data/{}_labels.csv'.format(directory), index=None)
        print('Successfully converted xml to csv.')


main()

import os

import glob

import pandas as pd

import xml.etree.ElementTree as ET

def xml_to_csv(path):

xml_list = []

for xml_file in glob.glob(path + '/*.xml'):

tree = ET.parse(xml_file)

root = tree.getroot()

for member in root.findall('object'):

value = (root.find('filename').text,

int(root.find('size')[0].text),

int(root.find('size')[1].text),

member[0].text,

int(member[4][0].text),

int(member[4][1].text),

int(member[4][2].text),

int(member[4][3].text)

)

xml_list.append(value)

column_name = ['filename', 'width', 'height', 'class', 'xmin', 'ymin', 'xmax', 'ymax']

xml_df = pd.DataFrame(xml_list, columns=column_name)

return xml_df

def main():

for directory in ['train','test']:

image_path = os.path.join(os.getcwd(), 'images/{}'.format(directory))

xml_df = xml_to_csv(image_path)

xml_df.to_csv('data/{}_labels.csv'.format(directory), index=None)

print('Successfully converted xml to csv.')

main()

In the above main function, you should specify your XML files path for both train and test folder. The generated CSV files will contain columns as filename, width, and height of images, output label of images and coordinates of the annotated rectangular box as shown in the figure below

Once you have your train and test images with labels in CSV format, let’s convert data in TFRecord format.

A TFRecord file store your data as a sequence of binary strings. It has many advantages over normal data formats. To do this we will use the following code which has been taken from this repository. According to your requirement, you need to change the condition for labels at line 31 below.

"""
Usage:
  # From tensorflow/models/
  # Create train data:
  python generate_tfrecord.py --csv_input=data/train_labels.csv  --output_path=train.record

  # Create test data:
  python generate_tfrecord.py --csv_input=data/test_labels.csv  --output_path=test.record
"""
from __future__ import division
from __future__ import print_function
from __future__ import absolute_import

import os
import io
import pandas as pd
import tensorflow as tf

from PIL import Image
from object_detection.utils import dataset_util
from collections import namedtuple, OrderedDict

flags = tf.app.flags
flags.DEFINE_string('csv_input', '', 'Path to the CSV input')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
flags.DEFINE_string('image_dir', '', 'Path to images')
FLAGS = flags.FLAGS


# TO-DO replace this with label map
def class_text_to_int(row_label):
    if row_label == 'up':
        return 1
    elif row_label == 'down':
        return 2
    elif row_label == 'left':
        return 3
    elif row_label == 'right':
        return 4
    else:
        None


def split(df, group):
    data = namedtuple('data', ['filename', 'object'])
    gb = df.groupby(group)
    return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]


def create_tf_example(group, path):
    with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:
        encoded_jpg = fid.read()
    encoded_jpg_io = io.BytesIO(encoded_jpg)
    image = Image.open(encoded_jpg_io)
    width, height = image.size

    filename = group.filename.encode('utf8')
    image_format = b'jpg'
    xmins = []
    xmaxs = []
    ymins = []
    ymaxs = []
    classes_text = []
    classes = []

    for index, row in group.object.iterrows():
        xmins.append(row['xmin'] / width)
        xmaxs.append(row['xmax'] / width)
        ymins.append(row['ymin'] / height)
        ymaxs.append(row['ymax'] / height)
        classes_text.append(row['class'].encode('utf8'))
        classes.append(class_text_to_int(row['class']))

    tf_example = tf.train.Example(features=tf.train.Features(feature={
        'image/height': dataset_util.int64_feature(height),
        'image/width': dataset_util.int64_feature(width),
        'image/filename': dataset_util.bytes_feature(filename),
        'image/source_id': dataset_util.bytes_feature(filename),
        'image/encoded': dataset_util.bytes_feature(encoded_jpg),
        'image/format': dataset_util.bytes_feature(image_format),
        'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
        'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
        'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
        'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
        'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
        'image/object/class/label': dataset_util.int64_list_feature(classes),
    }))
    return tf_example


def main(_):
    writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
    path = os.path.join(FLAGS.image_dir)
    examples = pd.read_csv(FLAGS.csv_input)
    grouped = split(examples, 'filename')
    for group in grouped:
        tf_example = create_tf_example(group, path)
        writer.write(tf_example.SerializeToString())

    writer.close()
    output_path = os.path.join(os.getcwd(), FLAGS.output_path)
    print('Successfully created the TFRecords: {}'.format(output_path))


if __name__ == '__main__':
    tf.app.run()

100

101

102

103

104

105

106

"""

Usage:

# From tensorflow/models/

# Create train data:

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record

# Create test data:

python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=test.record

"""

from __future__ import division

from __future__ import print_function

from __future__ import absolute_import

import os

import io

import pandas as pd

import tensorflow as tf

from PIL import Image

from object_detection.utils import dataset_util

from collections import namedtuple, OrderedDict

flags = tf.app.flags

flags.DEFINE_string('csv_input', '', 'Path to the CSV input')

flags.DEFINE_string('output_path', '', 'Path to output TFRecord')

flags.DEFINE_string('image_dir', '', 'Path to images')

FLAGS = flags.FLAGS

# TO-DO replace this with label map

def class_text_to_int(row_label):

if row_label == 'up':

return 1

elif row_label == 'down':

return 2

elif row_label == 'left':

return 3

elif row_label == 'right':

return 4

else:

None

def split(df, group):

data = namedtuple('data', ['filename', 'object'])

gb = df.groupby(group)

return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]

def create_tf_example(group, path):

with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:

encoded_jpg = fid.read()

encoded_jpg_io = io.BytesIO(encoded_jpg)

image = Image.open(encoded_jpg_io)

width, height = image.size

filename = group.filename.encode('utf8')

image_format = b'jpg'

xmins = []

xmaxs = []

ymins = []

ymaxs = []

classes_text = []

classes = []

for index, row in group.object.iterrows():

xmins.append(row['xmin'] / width)

xmaxs.append(row['xmax'] / width)

ymins.append(row['ymin'] / height)

ymaxs.append(row['ymax'] / height)

classes_text.append(row['class'].encode('utf8'))

classes.append(class_text_to_int(row['class']))

tf_example = tf.train.Example(features=tf.train.Features(feature={

'image/height': dataset_util.int64_feature(height),

'image/width': dataset_util.int64_feature(width),

'image/filename': dataset_util.bytes_feature(filename),

'image/source_id': dataset_util.bytes_feature(filename),

'image/encoded': dataset_util.bytes_feature(encoded_jpg),

'image/format': dataset_util.bytes_feature(image_format),

'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),

'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),

'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),

'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),

'image/object/class/text': dataset_util.bytes_list_feature(classes_text),

'image/object/class/label': dataset_util.int64_list_feature(classes),

}))

return tf_example

def main(_):

writer = tf.python_io.TFRecordWriter(FLAGS.output_path)

path = os.path.join(FLAGS.image_dir)

examples = pd.read_csv(FLAGS.csv_input)

grouped = split(examples, 'filename')

for group in grouped:

tf_example = create_tf_example(group, path)

writer.write(tf_example.SerializeToString())

writer.close()

output_path = os.path.join(os.getcwd(), FLAGS.output_path)

print('Successfully created the TFRecords: {}'.format(output_path))

if __name__ == '__main__':

tf.app.run()

Save this code in a file named generate_tfrecord.py. Now in order to use this code, first we need to clone tensorflow object detection API. For that do the following:

git clone https://github.com/tensorflow/models.git

1	git clone https://github.com/tensorflow/models.git

Then we need to do the following steps to avoid getting error of protoc:

Go to this release link and download protobuf according to your operating system.
Extract the downloaded file and go to bin folder inside it.
Copy protoc.exe file and put in models -> research -> object_detection -> protos folder.
In protos folder run the following command for .proto files.

protoc string_int_label_map.proto --python_out=.

1	protoc string_int_label_map.proto --python_out=.

After cloning this repository, copy generate_tfrecord.py inside models -> research folder and run the following command.

python generate_tfrecord.py --csv_input=data/train_labels.csv  --output_path=train.record --image_dir data/train

python generate_tfrecord.py --csv_input=data/test_labels.csv  --output_path=train.record --image_dir data/test

python generate_tfrecord.py --csv_input=data/train_labels.csv --output_path=train.record --image_dir data/train

python generate_tfrecord.py --csv_input=data/test_labels.csv --output_path=train.record --image_dir data/test

Above commands will generate two files named train.record and test.record which will be used for training of model.

This is all for generating TFRecord file, in the next blog we will perform training and testing of object detection model.

Next Blog: Snake Game Using Tensorflow Object Detection API – Part III

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Snake Game Using Tensorflow Object Detection API

1 Reply

Here, we will learn how to use tensorflow object detection API with the computer’s webcam to play a snake game. We will use hand gestures instead of the keyboard.

We will use following steps to play snake game using tensorflow object detection API:

Generate dataset.
Convert train and test datasets into tfrecord format.
Train a pre-trained model using generated data.
Integrate trained model with snake game.
Play the snake game using your own hand gestures.

In this blog, we will cover only the first step i.e. how to create your own training data and the rest steps will be covered in the subsequent blogs.

You can find code here.

Generate Dataset

A snake game problem generally contains four directions to move i.e. up, down, right and left. For each of the four directions, we need to generate at least 100 images per direction. You can use your phone or laptop camera to do this. Try to generate images with a different background for better generalization. Below are some examples of images of hand gestures.

Now we have our captured images of hand gestures. The next thing is to annotate these images according to their classes. Which means we need to create rectangular boxes around hand gestures and label them appropriately. Don’t worry there is a tool named LabelImg, which is highly helpful to annotate images to create training and test dataset. To start with LabelImg you can follow there GitHub link. The start screen of Labellmg would look like this.

At the left side of the screen, you can find various options. Click on Open dir and choose the input image folder. Then click on Change save dir and select output folder where generated XML files will be saved. This XML file will contain coordinates of your generated rectangular box in the image, something like this.

<annotation>
	<folder>Screenshots</folder>
	<filename>labellmg.png</filename>
	<path>\Pictures\Screenshots\labellmg.png</path>
	<source>
		<database>Unknown</database>
	</source>
	<size>
		<width>1920</width>
		<height>1080</height>
		<depth>3</depth>
	</size>
	<segmented>0</segmented>
	<object>
		<name>left</name>
		<pose>Unspecified</pose>
		<truncated>0</truncated>
		<difficult>0</difficult>
		<bndbox>
			<xmin>277</xmin>
			<ymin>234</ymin>
			<xmax>814</xmax>
			<ymax>780</ymax>
		</bndbox>
	</object>
</annotation>

<folder>Screenshots</folder>

<filename>labellmg.png</filename>

<path>\Pictures\Screenshots\labellmg.png</path>

<database>Unknown</database>

</source>

<size>

</size>

<pose>Unspecified</pose>

</bndbox>

</object>

</annotation>

To create a rectangular box in the image using Labellmg, you just need to press ‘W’ and then create a box and save it. You can create one or multiple boxes in one image as shown in the figure below. Repeat this for all the images.

Now we have images and their corresponding XML files. Then we will separate this dataset into training and testing in the 90/10 ratio. To do this we need to put 90% of images of each class ‘up’, ‘right’, ‘left’ and ‘down’ and their corresponding XML files in one folder and other 10% in other folder.

That’s all for creating dataset, in the next blog we will see how to create TFRecord files from these datasets which will be used for training of the model.

Next Blog: Snake Game Using Tensorflow Object Detection API – Part II

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

PEPs

Saving and Loading models in Keras

1 Reply

Generally, a deep learning model takes a large amount of time to train, so its better to know how to save trained model. In this blog we will learn about how to save whole keras model i.e. its architecture, weights and optimizer state.

Lets first create a model in Keras. This is a simple autoencoder model. If you need to know more about autoencoders please refer this blog.

from keras.models import Model
from keras.layers import Dense, Input
import matplotlib.pyplot as plt
import numpy as np

# creating an autoencoder model
input_shape = Input(shape = (784,))
dense1 = Dense(512, activation = 'relu')(input_shape)
dense2 = Dense(256, activation = 'relu')(dense1)
encoded = Dense(32, activation = 'relu')(dense2)

dense3 = Dense(256, activation = 'relu')(encoded)
dense4 = Dense(512, activation = 'relu')(dense3)
decoded = Dense(784, activation = 'relu')(dense4)

autoencoder = Model(input_shape, decoded)

m = 256
n_epoch = 25
autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.fit(output_X_train,output_X_train, epochs=n_epoch, batch_size=m, shuffle=True)

from keras.models import Model

from keras.layers import Dense, Input

import matplotlib.pyplot as plt

import numpy as np

# creating an autoencoder model

input_shape = Input(shape = (784,))

dense1 = Dense(512, activation = 'relu')(input_shape)

dense2 = Dense(256, activation = 'relu')(dense1)

encoded = Dense(32, activation = 'relu')(dense2)

dense3 = Dense(256, activation = 'relu')(encoded)

dense4 = Dense(512, activation = 'relu')(dense3)

decoded = Dense(784, activation = 'relu')(dense4)

autoencoder = Model(input_shape, decoded)

m = 256

n_epoch = 25

autoencoder.compile(optimizer='adam', loss='mse')

autoencoder.fit(output_X_train,output_X_train, epochs=n_epoch, batch_size=m, shuffle=True)

Above we have created a Keras model named as “autoencoder“. Now lets see how to save this model.

Saving and loading only architecture of a model

In keras, you can save and load architecture of a model in two formats: JSON or YAML Models generated in these two format are human readable and can be edited if needed.

# saving and loading model architecture in json format

# saving in json format
json_model = autoencoder.to_json()
json_file = open('autoencoder_json.json', 'w')
json_file.write(json_model)

# loading model architecture from json file
from keras.models import model_from_json
json_file = open('autoencoder_json.json', 'r')
json_model = model_from_json(json_file.read())

# saving and loading model architecture in json format

# saving in json format

json_model = autoencoder.to_json()

json_file = open('autoencoder_json.json', 'w')

json_file.write(json_model)

# loading model architecture from json file

from keras.models import model_from_json

json_file = open('autoencoder_json.json', 'r')

json_model = model_from_json(json_file.read())

# saving and loading model architecture in yaml format

# saving in yaml format
yaml_model = autoencoder.to_yaml()
yaml_file = open('autoencoder_yaml.yaml', 'w')
yaml_file.write(yaml_model)

# loading model architecture from yaml file
from keras.models import model_from_yaml
yaml_file = open('autoencoder_yaml.yaml', 'r')
yaml_model = model_from_yaml(yaml_file.read())

# saving and loading model architecture in yaml format

# saving in yaml format

yaml_model = autoencoder.to_yaml()

yaml_file = open('autoencoder_yaml.yaml', 'w')

yaml_file.write(yaml_model)

# loading model architecture from yaml file

from keras.models import model_from_yaml

yaml_file = open('autoencoder_yaml.yaml', 'r')

yaml_model = model_from_yaml(yaml_file.read())

Saving and Loading Weights of a Keras Model

With model architecture you will also need model weights to predict output from trained model.

# saving model weights
autoencoder.save_weights('autoencoder_weights.h5')

# loading weights of a keras model
json_model.load_weights('autoencoder_weights.h5')

# saving model weights

autoencoder.save_weights('autoencoder_weights.h5')

# loading weights of a keras model

json_model.load_weights('autoencoder_weights.h5')

Saving and Loading Both Architecture and Weights in one File

# saving whole model
autoencoder.save('autoencoder_model.h5')

# loading whole model
from keras.models import load_model
model1 = load_model('autoencoder_model.h5')

# saving whole model

autoencoder.save('autoencoder_model.h5')

# loading whole model

from keras.models import load_model

model1 = load_model('autoencoder_model.h5')

This will save following four parameters in “autoencoder_model.h5” file:

Model Architecture
Model Weights
Loss and Optimizer
State of the optimizer allowing to resume training where you left.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Sparse Autoencoders

1 Reply

In the last blog we have seen autoencoders and its applications. In this blog we will learn one of its variant, sparse autoencoders.

In every autoencoder, we try to learn compressed representation of the input. Let’s take an example of a simple autoencoder having input vector dimension of 1000, compressed into 500 hidden units and reconstructed back into 1000 outputs. The hidden units will learn correlated features present in the input. But what if input features are completely random? Then it will we difficult for hidden units to learn interesting structure present in data. In that situation what we can do is increase the number of hidden units and add some sparsity constraints. Now the question is what are sparsity constraints?

When sparsity constraints added to a hidden unit, it only activates some units (having large activation values) and makes rest to zero. So, even if we are having a large number of hidden units( as in the above example), it will only fire some hidden units and learn useful structure present in the data.

The simplest implementation of sparsity constraints can be done in keras. You can simple add activity_regularizer to a layer (see line 11) and it will do the rest.

from keras import regularizers
# creating autoencoder model
encoder_inputs = Input(shape = (28,28,1))

conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(encoder_inputs)
pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1)
conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1)
pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2)
flat = Flatten()(pool2)

enocder_outputs = Dense(32, activation = 'relu', activity_regularizer=regularizers.l1(10e-5))(flat)

#upsampling in decoder

dense_layer_d = Dense(7*7*32, activation = 'relu')(enocder_outputs)
output_from_d = Reshape((7,7,32))(dense_layer_d)
conv1_1 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(output_from_d)
upsampling_1 = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(conv1_1)
upsampling_2 = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(upsampling_1)
decoded_outputs = Conv2DTranspose(1, 3, padding='same', activation='relu')(upsampling_2)

autoencoder = Model(encoder_inputs, decoded_outputs)

from keras import regularizers

# creating autoencoder model

encoder_inputs = Input(shape = (28,28,1))

conv1 = Conv2D(16, (3,3), activation = 'relu', padding = "SAME")(encoder_inputs)

pool1 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv1)

conv2 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(pool1)

pool2 = MaxPooling2D(pool_size = (2,2), strides = 2)(conv2)

flat = Flatten()(pool2)

enocder_outputs = Dense(32, activation = 'relu', activity_regularizer=regularizers.l1(10e-5))(flat)

#upsampling in decoder

dense_layer_d = Dense(7*7*32, activation = 'relu')(enocder_outputs)

output_from_d = Reshape((7,7,32))(dense_layer_d)

conv1_1 = Conv2D(32, (3,3), activation = 'relu', padding = "SAME")(output_from_d)

upsampling_1 = Conv2DTranspose(32, 3, padding='same', activation='relu', strides=(2, 2))(conv1_1)

upsampling_2 = Conv2DTranspose(16, 3, padding='same', activation='relu', strides=(2, 2))(upsampling_1)

decoded_outputs = Conv2DTranspose(1, 3, padding='same', activation='relu')(upsampling_2)

autoencoder = Model(encoder_inputs, decoded_outputs)

But, if you want to add sparse constraints by writing your own function, you can follow reference given below.

References: Sparse Autoencoders

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

TheAILearner

Mastering Artificial Intelligence

Tag Archives: python

Show current DateTime on live video using OpenCV-Python

Steps:

Code:

Scraping Video Information from YouTube

Creating a Snake Game using OpenCV-Python

Import Libraries

Displaying Game Objects

Game Rules

Displaying the final Score

Snake Game Using Tensorflow Object Detection API – Part IV

Snake Game Using Tensorflow Object Detection API – Part III

Snake Game Using Tensorflow Object Detection API – Part II

Snake Game Using Tensorflow Object Detection API

Generate Dataset

PEPs

Saving and Loading models in Keras

Sparse Autoencoders