2D histogram

In this blog, we will discuss Histogram Backprojection, a technique that is used for image segmentation or finding objects of interest in an image. It was proposed by Michael J. Swain, Dana H. Ballard in their paper Indexing via color histograms, Third international conference on computer vision,1990.

This was one of the first works that use color to address the classic problem of Classification and Localisation in computer vision.

To understand this technique, knowledge of histograms (particularly 2-D histograms) is a must. If you haven’t encountered 2-d histograms yet, I suggest you to read What is a 2-D histogram?

Now, let’s see what is Histogram backprojection and how do we do it?

According to the authors, Histogram Backprojection answers the question

“Where are the colors in the image that belong to the object being looked for (the target)?”

So, this addresses the localization problem i.e. where is the object in an image. In this, we calculate the histogram model of a feature and then use it to find this feature in an image. To know why this is named as Histogram Backprojection, you need to know how this method works. So, let’s see how to do this

Suppose we want to find the green color in the target image. Let ‘roi’ be the image of the object we need to find and ‘target’ be the image where we are going to search that object.

Steps:

First, load the images, convert them into HSV and find the histograms as shown below

import cv2
import numpy as np
import matplotlib.pyplot as plt
#roi is the object or region of object we need to find
roi = cv2.imread('D:/downloads/messi_ground.jpg')
hsv = cv2.cvtColor(roi,cv2.COLOR_BGR2HSV)
#target is the image we search in
target = cv2.imread('D:/downloads/messi.jpg')
hsvt = cv2.cvtColor(target,cv2.COLOR_BGR2HSV)
# Find the histograms using calcHist.
M = cv2.calcHist([hsv],[0, 1], None, [180, 256], [0, 180, 0, 256] )
I = cv2.calcHist([hsvt],[0, 1], None, [180, 256], [0, 180, 0, 256] )

import cv2

import numpy as np

import matplotlib.pyplot as plt

#roi is the object or region of object we need to find

roi = cv2.imread('D:/downloads/messi_ground.jpg')

hsv = cv2.cvtColor(roi,cv2.COLOR_BGR2HSV)

#target is the image we search in

target = cv2.imread('D:/downloads/messi.jpg')

hsvt = cv2.cvtColor(target,cv2.COLOR_BGR2HSV)

# Find the histograms using calcHist.

M = cv2.calcHist([hsv],[0, 1], None, [180, 256], [0, 180, 0, 256] )

I = cv2.calcHist([hsvt],[0, 1], None, [180, 256], [0, 180, 0, 256] )

The 2-D histograms looks like this

This was expected, as roi image has shades of green so its histogram is mostly concentrated to the H and S values representing green. Similarly we can argue for the target image.

Now, what we want is to make ‘I’ look as similar to ‘M‘ as possible. Only then we will be able to extract the green color from the target image. So, let’s see how to do this.

One plausible solution is to divide “M” by “I”. This way the output will have values greater than 0, where both “M” and “I” are greater than 0. For all other cases, it will be either ‘0’ or ‘Nan’.

R = M/I

R = M/I

The output ‘R’ is shown below where cyan represents values greater than 0 (probably our roi), purple represents 0 and white represents Nan.

Now the last thing to do is, find the pixels in the target image that corresponds to the cyan region shown above. In other words, we back project the 2-D histogram.

Because we know the “H” and “S” values for the cyan region (See R image above), we can easily find out the pixels with similar “H” and “S” values in the target image. Let’s see how to do this.

First, we will extract “H” and “S” channels from the target image.

h,s,v = cv2.split(hsvt)

1	h,s,v = cv2.split(hsvt)

For each pixel in the target image, using the “H” and “S” value for that pixel we will find the corresponding value in the 2-D histogram and save that value in ‘B’.

B = R[h.ravel(),s.ravel()]

1	B = R[h.ravel(),s.ravel()]

Note: “B” doesn’t contain “Nan” values. Remember, “Nan” occurs when both “M” and “I” equals to 0.

We keep the values between 0 and 1 so that the value can be treated as the probability of each pixel belonging to the target. After that, we resize “B”.

B = np.minimum(B,1)
B = B.reshape(hsvt.shape[:2])

1 2	B = np.minimum(B,1) B = B.reshape(hsvt.shape[:2])

So, now we have created a new image “B” (size same as that of the target image) where every pixel value represents the corresponding probability of being the target. Brighter pixels are more probable of being the target. “B” is shown below

Now, the next step is just for fine-tuning this output. This varies from image to image.

# apply a convolution with a circular disc
disc = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5,5))
cv2.filter2D(B,-1,disc,B)
B = np.uint8(B)
cv2.normalize(B,B,0,255,cv2.NORM_MINMAX)

# apply a convolution with a circular disc

disc = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5,5))

cv2.filter2D(B,-1,disc,B)

B = np.uint8(B)

cv2.normalize(B,B,0,255,cv2.NORM_MINMAX)

Use thresholding to segment out the region and Overlay images using bitwise_and to produce the desired output.

# Use thresholding to segment out the region
ret,thresh = cv2.threshold(B,10,255,0)

# Overlay images using bitwise_and
thresh = cv2.merge((thresh,thresh,thresh))
res = cv2.bitwise_and(target,thresh)

# Display the output
cv2.imshow('a',res)
cv2.waitKey(0)

# Use thresholding to segment out the region

ret,thresh = cv2.threshold(B,10,255,0)

# Overlay images using bitwise_and

thresh = cv2.merge((thresh,thresh,thresh))

res = cv2.bitwise_and(target,thresh)

# Display the output

cv2.imshow('a',res)

cv2.waitKey(0)

The output looks like this

See how from a 2-D histogram we are able to extract the roi from the target image.

Backprojection in OpenCV

OpenCV provides an inbuilt function

cv2.calcBackProject( target_img, channels, roi_hist, ranges, scale )

target_img: image where you want to find the feature.
channels: The list of channels used to compute the back projection.
roi_hist: histogram of the feature you want to find.
ranges: histogram bin boundaries in each dimension.
scale: Optional scale factor for the output back projection.

This returns the probability image “B”.

So, we only need to calculate the roi histogram (M) and normalize it. No need of calculating “I” and “R”. This function directly output “B”.

# calculating object histogram
M = cv2.calcHist([hsv],[0, 1], None, [180, 256], [0, 180, 0, 256] )

# normalize histogram and apply backprojection
cv2.normalize(M,M,0,255,cv2.NORM_MINMAX)
B = cv2.calcBackProject([hsvt],[0,1],M,[0,180,0,256],1)

# calculating object histogram

M = cv2.calcHist([hsv],[0, 1], None, [180, 256], [0, 180, 0, 256] )

# normalize histogram and apply backprojection

cv2.normalize(M,M,0,255,cv2.NORM_MINMAX)

B = cv2.calcBackProject([hsvt],[0,1],M,[0,180,0,256],1)

After that apply all the fine-tuning steps that we did earlier.

That’s all about Histogram Backprojection. Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time

In the last blog, we discussed 1-D histograms in which we analyze each channel separately. Suppose, we want to find the correlation between image channels, let’s say we are interested in finding like how many times a (red, green) pair of (100,56) appeared in an image. In such a case, a 1-D histogram will fail as it does not shows the relationship of intensities at the exact position between two channels.

To solve this problem, we need Multi-dimensional histograms like 2-D or 3D. With the help of 2-D histograms, we can analyze the channels together in groups of 2 (RG, GB, BR) or all together with 3D histograms. Let’s see what is a 2-D histogram and how to construct this using OpenCV Python.

A 2-D histogram counts the occurrence of combinations of intensities. Below figure shows a 2D histogram

Here, Y and X-axis correspond to the Red and Green channel ranges( for 8-bit, [0,255]) and each point within the histogram shows the frequency corresponding to each R and G pair. Frequency is color-coded here, otherwise, another dimension would be needed.

Let’s understand how to construct a 2-D histogram by taking a simple example.

Suppose, we have 4×4, 2-bit images of Red and Green channels(as shown below) and we want to plot their 2-D histogram.

First, we plot the R and G channel ranges(Here, [0,3]) on the X and Y-axis respectively. This will be our 2-D histogram.

Then, loop over each position within the channels, find the corresponding intensity pairs frequency and plot it in the 2-D histogram. These frequencies are then color-coded for ease of visualization.

Now, let’s see how to construct a 2-D histogram using OpenCV-Python

We use the same function cv2.calcHist() that we have used for a 1-D histogram. Just change the following parameters and rest is the same.

channels: [0,1] for (Blue, Green), [1,2] for (G, R) and [0,2] for (B, R).
bins: specify for each channel according to your need. e.g [256,256].
range: [0,256,0,256] for an 8-bit image.

Below is the sample code for this using OpenCV-Python

# 2D histogram for Blue and Green channels.
hist = cv2.calcHist([image], [0, 1], None, [256, 256], [0, 256, 0, 256])
# show using matplotlib
plt.imshow(hist, interpolation = 'nearest')
plt.show()

# 2D histogram for Blue and Green channels.

hist = cv2.calcHist([image], [0, 1], None, [256, 256], [0, 256, 0, 256])

# show using matplotlib

plt.imshow(hist, interpolation = 'nearest')

plt.show()

Always use Nearest Neighbour Interpolation when plotting a 2-D histogram.

Plotting a 2-D histogram using RGB channels is not a good choice as we cannot extract color information using 2 channels only. Still, this can be used for finding the correlation between channels, finding clipping or intensity proportions etc.

To extract color information, we need a color model in which two components/channels can solely represent the chromaticity (color) of the image. One such color model is HSV where H and S tell us about the color of the light. So, first convert the image from BGR to HSV and then apply the above code.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

TheAILearner

Mastering Artificial Intelligence

Tag Archives: 2D histogram

Histogram Backprojection

Steps:

Backprojection in OpenCV