Tag Archives: opencv

Creating a CRNN model to recognize text in an image (Part-1)

In the earlier blogs, we learned various stages of optical character recognition pipeline. In this blog, we will create a convolutional recurrent neural network with CTC (Connectionist Temporal Classification) loss to implement our recognition model.

We will use the following steps to create our text recognition model.

Collecting Dataset
Preprocessing Data
Creating Network Architecture
Defining Loss function
Training model
Decoding outputs from prediction

Dataset

In this blog, we will use data provided by Visual Geometry Group. This is a huge dataset total of 10 GB images. Here I have used only 135000 images for the training set and 15000 images for validation dataset. This data contains text image segments which look like images shown below:

Source

To download the dataset either you can directly download from this link or use the following commands to download the data and unzip.

wget https://www.robots.ox.ac.uk/~vgg/data/text/mjsynth.tar.gz

tar -xvzf mjsynth.tar.gz

wget https://www.robots.ox.ac.uk/~vgg/data/text/mjsynth.tar.gz

tar -xvzf mjsynth.tar.gz

Preprocessing

Now we are having our dataset, to make it acceptable for our model we need to use some preprocessing. We need to preprocess both the input image and output labels. To preprocess our input image we will use followings:

Read the image and convert into a gray-scale image
Make each image of size (128,32) by using padding
Expand image dimension as (128,32,1) to make it compatible with the input shape of architecture
Normalize the image pixel values by dividing it with 255.

To preprocess the output labels use the followings:

Read the text from the name of the image as the image name contains text written inside the image.
Encode each character of a word into some numerical value by creating a function( as ‘a’:0, ‘b’:1 …….. ‘z’:26 etc ). Let say we are having the word ‘abab’ then our encoded label would be [0,1,0,1]
Compute the maximum length from words and pad every output label to make it of the same size as the maximum length. This is done to make it compatible with the output shape of our RNN architecture.

In preprocessing step we also need to create two other lists: one is label length and other is input length to our RNN. These two lists are important for our CTC loss( we will see later ). Label length is the length of each output text label and input length is the same for each input to the LSTM layer which is 31 in our architecture.

Note: For more details on the Optical Character Recognition , please refer to the Mastering OCR using Deep Learning and OpenCV-Python course.

Following is the code for our preprocessing step:

# char_list:   'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
# total number of our output classes: len(char_list)
char_list = string.ascii_letters+string.digits

def encode_to_labels(txt):
    # encoding each output word into digits
    dig_lst = []
    for index, char in enumerate(txt):
        try:
            dig_lst.append(char_list.index(char))
        except:
            print(char)
        
    return dig_lst

# char_list: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'

# total number of our output classes: len(char_list)

char_list = string.ascii_letters+string.digits

def encode_to_labels(txt):

# encoding each output word into digits

dig_lst = []

for index, char in enumerate(txt):

try:

dig_lst.append(char_list.index(char))

except:

print(char)

return dig_lst

path = '/mnt/ramdisk/max/90kDICT32px'


# lists for training dataset
training_img = []
training_txt = []
train_input_length = []
train_label_length = []
orig_txt = []

#lists for validation dataset
valid_img = []
valid_txt = []
valid_input_length = []
valid_label_length = []
valid_orig_txt = []

max_label_len = 0

i =1 
flag = 0

for root, dirnames, filenames in os.walk(path):

    for f_name in fnmatch.filter(filenames, '*.jpg'):
        # read input image and convert into gray scale image
        img = cv2.cvtColor(cv2.imread(os.path.join(root, f_name)), cv2.COLOR_BGR2GRAY)   

        # convert each image of shape (32, 128, 1)
        w, h = img.shape
        if h > 128 or w > 32:
            continue
        if w < 32:
            add_zeros = np.ones((32-w, h))*255
            img = np.concatenate((img, add_zeros))

        if h < 128:
            add_zeros = np.ones((32, 128-h))*255
            img = np.concatenate((img, add_zeros), axis=1)
        img = np.expand_dims(img , axis = 2)
        
        # Normalize each image
        img = img/255.
        
        # get the text from the image
        txt = f_name.split('_')[1]
        
        # compute maximum length of the text
        if len(txt) > max_label_len:
            max_label_len = len(txt)
            
           
        # split the 150000 data into validation and training dataset as 10% and 90% respectively
        if i%10 == 0:     
            valid_orig_txt.append(txt)   
            valid_label_length.append(len(txt))
            valid_input_length.append(31)
            valid_img.append(img)
            valid_txt.append(encode_to_labels(txt))
        else:
            orig_txt.append(txt)   
            train_label_length.append(len(txt))
            train_input_length.append(31)
            training_img.append(img)
            training_txt.append(encode_to_labels(txt)) 
        
        # break the loop if total data is 150000
        if i == 150000:
            flag = 1
            break
        i+=1
    if flag == 1:
        break
        
# pad each output label to maximum text length

train_padded_txt = pad_sequences(training_txt, maxlen=max_label_len, padding='post', value = len(char_list))
valid_padded_txt = pad_sequences(valid_txt, maxlen=max_label_len, padding='post', value = len(char_list))

path = '/mnt/ramdisk/max/90kDICT32px'

# lists for training dataset

training_img = []

training_txt = []

train_input_length = []

train_label_length = []

orig_txt = []

#lists for validation dataset

valid_img = []

valid_txt = []

valid_input_length = []

valid_label_length = []

valid_orig_txt = []

max_label_len = 0

i =1

flag = 0

for root, dirnames, filenames in os.walk(path):

for f_name in fnmatch.filter(filenames, '*.jpg'):

# read input image and convert into gray scale image

img = cv2.cvtColor(cv2.imread(os.path.join(root, f_name)), cv2.COLOR_BGR2GRAY)

# convert each image of shape (32, 128, 1)

w, h = img.shape

if h > 128 or w > 32:

continue

if w < 32:

add_zeros = np.ones((32-w, h))*255

img = np.concatenate((img, add_zeros))

if h < 128:

add_zeros = np.ones((32, 128-h))*255

img = np.concatenate((img, add_zeros), axis=1)

img = np.expand_dims(img , axis = 2)

# Normalize each image

img = img/255.

# get the text from the image

txt = f_name.split('_')[1]

# compute maximum length of the text

if len(txt) > max_label_len:

max_label_len = len(txt)

# split the 150000 data into validation and training dataset as 10% and 90% respectively

if i%10 == 0:

valid_orig_txt.append(txt)

valid_label_length.append(len(txt))

valid_input_length.append(31)

valid_img.append(img)

valid_txt.append(encode_to_labels(txt))

else:

orig_txt.append(txt)

train_label_length.append(len(txt))

train_input_length.append(31)

training_img.append(img)

training_txt.append(encode_to_labels(txt))

# break the loop if total data is 150000

if i == 150000:

flag = 1

break

i+=1

if flag == 1:

break

# pad each output label to maximum text length

train_padded_txt = pad_sequences(training_txt, maxlen=max_label_len, padding='post', value = len(char_list))

valid_padded_txt = pad_sequences(valid_txt, maxlen=max_label_len, padding='post', value = len(char_list))

Now you might have got some feeling about the training and validation data generation for our recognition model. In the next blog, we will use this data to train and test our neural network.

Next Blog: Creating a CRNN model to recognize text in an image (Part-2)

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Image Enhancement

Leave a reply

Till now, we learned the basics of an image. From now onwards, we will learn what actually is known as image processing. In this blog, we will learn what is image enhancement, different methods to perform image enhancement and then we will learn how we can perform this on real images.

According to MathWorks, Image enhancement is the process of adjusting digital images so that the results are more suitable for display or further image analysis. It is basically a preprocessing step.

Image enhancement can be done either in the spatial domain or transform domain. Spatial domain means we perform all operations directly on pixels while in transform domain we first transform an image into another domain (like frequency) do processing there and convert it back to the spatial domain by some inverse operations. We will be discussing these in detail in the next blogs.

Both spatial and transform domain have their own importance which we will discuss later. Generally, operations in spatial domain are more computationally efficient.

Processing in spatial domain can be divided into two main categories – one that operates on single pixels known as Intensity transformation and other known as Spatial filtering that works on the neighborhood of every pixel

The following example will motivate you about what we are going to study in the next few blogs

In the next blog, we will discuss how basic arithmetic operations like addition, subtraction etc can be used for image enhancement. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Color Models

Leave a reply

In the previous blogs, we represented the color image using the RGB components but this is not the only way available. There are different color models ( A color model is simply a way to define the color) available, each having their own pros and cons.

There are two types of color models available: Additive and Subtractive. Additive uses light (transmitted) to display color while subtractive models use printing inks. These models are fitted into different shapes to obtain new models (See HSI model below).

In this blog, we’ll discuss the three that are most commonly used in the context of digital image processing: RGB, CMY, and HSI

The RGB Color Model

In this, we construct a color cube whose 3 axes denote R, G, and B respectively as shown below

Normalized RGB Color Cube; Source: Researchgate

This is an additive model, i.e. the colors present in the light add to form new colors. For example, Yellow has coordinate of (1,1,0) which means Yellow = Red + Green. Similarly, for other colors like cyan = Blue + Green and magenta = Red +Blue.

R, G, and B are added together in varying proportions to produce an extensive range of colors. Mixing equal proportions of R, G, and B falls on the grayscale line.

Use: color monitors and most video cameras.

The CMYK Color Model

CMY stands for cyan, magenta, and yellow also known as secondary colors of light. K refers to black. An equal proportion of C, M, and Y produce muddly black and not pure black. That’s why we use CMYK instead of CMY model.

This is a subtractive model i.e colors are perceived as a result of reflected light. e.g. when light falls on a cyan coated surface, red is absorbed (or subtracted) while Green and Blue are reflected and thus G + B = Cyan. Similarly for magenta and yellow.

Thus, CMY can be obtained from RGB by subtracting RGB from the max intensity.

Use: Printing like books, magazines etc.

The HSI Color Model

HSI stands for Hue, Saturation, and Intensity. This model is similar to how humans perceive color. Let’s understand HSI terms

Hue: Color attribute that describes the pure color or dominant wavelength.

Saturation: Purity of Color or how much a pure color is diluted by white light.

Intensity: Amount of light

H and S tell us about the chromaticity (color information) of the light while I carries the greyscale information.

HSI model can be obtained by rotating the RGB cube such that Black is at the bottom and white at the top.

H varies from 0 to 120 degrees for Red, 120 – 240 for Green, and 240 -360 for Blue. Saturation can take value from 0 to 100%. Intensity value varies according to the bit size of an image.

Pros: Easier to represent the color than the RGB model.

Note: We can also use these color models for object tracking (See here).

In the next blog, we will see how different colors can be generated from these color models with the help of OpenCV. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Changing Video Resolution using OpenCV-Python

3 Replies

In this tutorial, I will show how to change the resolution of the video using OpenCV-Python. This blog is based on interpolation methods (Chapter-5) which we have discussed earlier.

Here, I will convert a 640×480 video to 1280×720. Let’s see how to do this

Steps:

Load a video using cv2.VideoCapture()
Create a VideoWriter object using cv2.VideoWriter()
Extract frame by frame
Resize the frames using cv2.resize()
Save the frames to a video file using cv2.VideoWriter()
Release the VideoWriter and destroy all windows

Code:

import cv2
import numpy as np

cap = cv2.VideoCapture('C:/New folder/video.avi')

fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('output.avi',fourcc, 5, (1280,720))

while True:
    ret, frame = cap.read()
    if ret == True:
        b = cv2.resize(frame,(1280,720),fx=0,fy=0, interpolation = cv2.INTER_CUBIC)
        out.write(b)
    else:
        break
    
cap.release()
out.release()
cv2.destroyAllWindows()

import cv2

import numpy as np

cap = cv2.VideoCapture('C:/New folder/video.avi')

fourcc = cv2.VideoWriter_fourcc(*'XVID')

out = cv2.VideoWriter('output.avi',fourcc, 5, (1280,720))

while True:

ret, frame = cap.read()

if ret == True:

b = cv2.resize(frame,(1280,720),fx=0,fy=0, interpolation = cv2.INTER_CUBIC)

out.write(b)

else:

break

cap.release()

out.release()

cv2.destroyAllWindows()

Here, I have used Bicubic as the interpolation method, you can use any. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Bayer filter

Leave a reply

In the previous blog, we saw different methods by which color image can be obtained from an image sensor. Out of these methods, the Bayer filter is most widely used today and in this blog, we will discuss it in detail.

To form a color image, we need to collect information at RGB wavelengths for all the pixels or sensors. But this process is expensive both in terms of time and money.

So in 1976, Bayer thought of an alternative. Instead of capturing all RGB information at each pixel, Bayer thought of capturing one out of RGB for each pixel. Now, each pixel will contain either R, G or B. To be able to form a color image, he decided 50% pixels be Green and rest equally to Red and Blue (to mimic human eye) and these are arranged in a pattern as shown below

He would then use interpolation or color demosaicing algorithm to find the missing information for example pixel capturing Red will need Green and Blue and so on. We will study in more detail about interpolation algorithms in next blog.

The overall procedure from Bayer to RGB color image can be summarized as

Source: Thesis

So, with Bayer filter, we are only storing one color information(either R, G or B) at each pixel which reduces the computation time and cost while maintaining the image quality. That’s why it is used widely.

Hope you understand the Bayer filter, why it is used and how the color image is obtained from the Bayer image.Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Understanding Images with OpenCV-Python

Leave a reply

In the previous blogs, we learned about pixels, intensity value, color, and greyscale image. Now, with OpenCV and numpy, let’s try to visualize these concepts.

First, we need an image, either you can load one or can make own image. Loading an image from the device looks like this

import cv2
import numpy as np

img = cv2.imread('F:/downloads/Python.png')

import cv2

import numpy as np

img = cv2.imread('F:/downloads/Python.png')

To access the pixel location, we must first know the shape of the image. This can be done by

>>> img.shape
(2000, 2000, 3)

1 2	>>> img.shape (2000, 2000, 3)

It returns a tuple of the number of rows, columns, and channels (if the image is color).

Total number of pixels can be found either by multiplying rows, columns, channels found using img.shape or by using the following command

>>> img.size          # same as 2000*2000*3
12000000

1 2	>>> img.size # same as 200020003 12000000

After knowing the image shape, we can access the pixel location by its row and column coordinates as

>>> img[250,250]                     # for single pixel
>>> img[280:340, 330:390]            # for image region

1 2	>>> img[250,250] # for single pixel >>> img[280:340, 330:390] # for image region

This returns the intensity value at that pixel location. For a greyscale image, intensity or pixel value is a single integer while for a color image, it is an array of Blue, Green, Red values.

# for color image, [Blue,Green,Red] value is returned
>>> img[250,250]
[157 166 200]
# for 8-bit greyscale image
>>> img[250,250]
257

# for color image, [Blue,Green,Red] value is returned

>>> img[250,250]

[157 166 200]

# for 8-bit greyscale image

>>> img[250,250]

257

Note: OpenCV reads the color image in BGR mode and not in RGB mode. Be careful

We know that intensity levels depend on the number of bits that can be found by

>>> img.dtype
dtype('uint8')           # 8-bit image

1 2	>>> img.dtype dtype('uint8') # 8-bit image

To access the RGB channels separately, use numpy indexing as shown below

>>> img[:,:,0]          # 0 for Blue, 1 - Green, 2 - Red

1	>>> img[:,:,0] # 0 for Blue, 1 - Green, 2 - Red

You can change the pixel value just by normal assignment as shown below

>>> img[250,250]
[0,0,0]
>>> img[250,250] = 34
>>> img[250,250]
[34,34,34]
# we can assign different values to each channel at a pixel
>>> img[250,250] = [234,145,23]
# to make all the red pixels to zero 
>>> img[:,:,2] = 0

>>> img[250,250]

[0,0,0]

>>> img[250,250] = 34

>>> img[250,250]

[34,34,34]

# we can assign different values to each channel at a pixel

>>> img[250,250] = [234,145,23]

# to make all the red pixels to zero

>>> img[:,:,2] = 0

You can change the color image to greyscale using the following command

grey = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

1	grey = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

All the operations that you can perform on the array like add, subtract etc apply to images also.

Play with all these commands to understand better. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Read, Write and Display Videos with OpenCV Python

Leave a reply

In this blog, we will see how to Read, Write and Display Videos using OpenCV. Since a video is made up of images, most of the commands we learned in the previous blog also applies here.

Let’s see by an example how to capture video from the camera and display it.

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
    ret, frame = cap.read()
    
    cv2.imshow('frame',frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

import numpy as np

import cv2

cap = cv2.VideoCapture(0)

while(True):

ret, frame = cap.read()

cv2.imshow('frame',frame)

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

cv2.VideoCapture(0) will open the default camera. You can select the second camera by passing 1, third by passing 2 and so on. This creates a VideoCapture object (“cap” here).

cap.read() capture frame by frame. This returns two values, frame and ret. If the frame is read correctly, ret will be True otherwise False.

cv2.waitKey(1) & 0xFF == ord(‘q’) will exit the video when ‘q’ is pressed.

cap.release() closes video file or capturing device.

If you want to play a video from a file, just change the cv2.VideoCapture(0) function in the above code by giving the file path as cv2.VideoCapture(‘F:/downloads/Python.mp4’). Also, use the appropriate time for cv2.waitKey() function (25 will be OK).

Saving a Video:

First, create a VideoWriter object with cv2.VideoWriter(output filename, fourcc, fps, frameSize). fourcc(4-character code of codec) is used to compress the frames. After creating this object, use object.write() to save the video. Let’s see an example

import numpy as np
import cv2

cap = cv2.VideoCapture(0)
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('output.avi',fourcc, 20.0, (640,480))
while(True):
    ret, frame = cap.read()
    cv2.imshow('frame',frame)
    out.write(frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

import numpy as np

import cv2

cap = cv2.VideoCapture(0)

fourcc = cv2.VideoWriter_fourcc(*'XVID')

out = cv2.VideoWriter('output.avi',fourcc, 20.0, (640,480))

while(True):

ret, frame = cap.read()

cv2.imshow('frame',frame)

out.write(frame)

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

Now, you might have got some feeling about the basic video commands in OpenCV. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Read, Write and Display Images with OpenCV

Leave a reply

In this blog, we will see how to Read, Write and Display Images using OpenCV. I hope you have installed OpenCV, numpy, and matplotlib libraries, if not, please refer to this blog.

Read an image:

To read an image, use the function cv2.imread(filename[, flags]) where filename is the full path of image and flags specifies the way image should be read (>0 for color, =0 for greyscale, and <0 for loading image as is (with alpha channel)).

import numpy as np
import cv2
# Load an image in grayscale
img = cv2.imread('F:/downloads/Python.png',0)

import numpy as np

import cv2

# Load an image in grayscale

img = cv2.imread('F:/downloads/Python.png',0)

If the image cannot be read (because of missing file, improper permissions, unsupported or invalid format), the function returns an empty matrix, not an error.

Display an image:

To display an image in a window, use the function cv2.imshow(winname, image) where the first argument is the Name of the window and second is the Image to be shown. So, this will first create a window named as image and displays the image in that window.

Note: This function must be followed by cv2.waitkey(delay) function otherwise the image wouldn’t be displayed.

cv2.imshow('image',img)
cv2.waitKey(0)
cv2.destroyAllWindows()

cv2.imshow('image',img)

cv2.waitKey(0)

cv2.destroyAllWindows()

cv2.waitKey(delay) decides for how long the image will be displayed. Its argument delay is the time in milliseconds. If the delay is <=0, the image will be shown forever otherwise destroyed after delay milliseconds.

cv2.destroyAllWindows() simply destroys all the windows we created.

Special Case: We can create a window first and load the image to it later. Just write the below code line before the cv2.imshow() function.

cv2.namedWindow('image', cv2.WINDOW_NORMAL)

1	cv2.namedWindow('image', cv2.WINDOW_NORMAL)

Write an image:

To save an image, use the function cv2.imwrite(filename, image) where the first argument is the file name with which we want to save the image file, the second argument is the image you want to save.

cv2.imwrite('pythongray.jpg',img)

1	cv2.imwrite('pythongray.jpg',img)

This will save the image in JPEG format in the working directory.

Now, you might have got some feeling about the basic image commands in OpenCV. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Installing Python OpenCV and other libraries

1 Reply

Here, we will be installing the libraries that are required to perform image processing operations. We will be using the following Python libraries

OpenCV
Numpy
Matplotlib

Why we are using OpenCV, not Matlab or any other?

Because it is open source, fast(written in C/C++), memory efficient and easy to install (can run on any device that can run C).
if you really want to learn about how computer vision works from the initial steps to the last, I suggest you learn OpenCV first then use any deep learning library like Tensorflow, pytorch etc when you’re ready to train a deep learning algorithm for better performance/accuracy results.

Installing Numpy, OpenCV or any library (For WINDOWS)

There are two ways to install any library in Python IDLE,

using pip command: In the installed Python folder, go to Scripts folder and open command prompt(press and hold Shift + Right Click and select Open command window here). Then write pip install library name to get it installed. For example

Python

pip install numpy

1

pip install numpy
using .whl file: First download .whl file of any library (version corresponding to your Python) from here. Then open the command prompt where you have downloaded this file and write pip install filename.whl. For example

Python

pip install numpy‑1.14.2+mkl‑cp36‑cp36m‑win32.whl

1

pip install numpy‑1.14.2+mkl‑cp36‑cp36m‑win32.whl

Installing Numpy, OpenCV or any library (For UBUNTU)

First, install pip using apt-get, then you can install any library as shown below

sudo apt-get install python-pip  
sudo pip install numpy

1 2	sudo apt-get install python-pip sudo pip install numpy

Anaconda: open the Anaconda prompt and write pip install numpy or any other library name which you want to install.

Now, you might have got some feeling about how to install Python libraries. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

TheAILearner

Mastering Artificial Intelligence

Tag Archives: opencv

Creating a CRNN model to recognize text in an image (Part-1)

Dataset

Preprocessing

Image Enhancement

Like this:

Color Models

The RGB Color Model

The CMYK Color Model

The HSI Color Model

Changing Video Resolution using OpenCV-Python

Steps:

Code:

Bayer filter

Understanding Images with OpenCV-Python

Read, Write and Display Videos with OpenCV Python

Read, Write and Display Images with OpenCV

Read an image:

Display an image:

Write an image:

Installing Python OpenCV and other libraries