Tag Archives: EAST

Implementation of EAST

Leave a reply

In the previous blog, we discussed the theory behind the EAST algorithm. If you remember, we stated that this algorithm is both accurate and efficient. So, in this blog, let’s find it out. For this, first, we will run the EAST algorithm using its Github repository, and then we will analyze the results. So, let’s get started. Here, I’m using a Linux system.

Clone the Repository

First, search “EAST Github” in the browser. You will find several EAST implementations but in this blog, we will use the one provided by argman. So, open this and clone the repository. In order to clone the repository, you can either use git or download it as a zip file. To install git, you can run the following command.

Once you have installed git, clone the repository using the following command.

This will clone the repository into your system as shown below.

Compile lanms

As you might remember, in the previous blog, we discussed that the EAST algorithm uses a Locality-Aware NMS (lanms) instead of the standard NMS. Now, you need to compile the lanms. Why? because this GitHub implementation contains the lanms code written in C++ (See the lanms folder). So, in order to make it work with Python, we need to generate an adaptor.so file. This can be done as follows.

First, we need to install the g++ compiler in order to compile the adaptar.cpp file. This can be done using the following command.

This contains the essential tools for building most other packages from source (e.g. C/C++ compiler, libc, and make).

Note: For more details on the Optical Character Recognition , please refer to the Mastering OCR using Deep Learning and OpenCV-Python course.

Next, open the __init__.py file present inside the lanms folder and comment out the if condition as shown below.

Again open the terminal and change the directory to the lanms folder. After this, run the make command as shown below. This will generate the required adaptor.so file in the lanms folder.

Test the model

Now, to test the model, either we need to first train it or find some pre-trained weights, if available. Luckily, pre-trained weights are available. You can download it from here. These are trained on the ICDAR-2013 and ICDAR-2015 datasets.

After downloading the pre-trained weights, extract them and place them inside the EAST folder. Now, to test the model, open the terminal and change the directory to the EAST folder. Also activate the virtual environment if any. Then type the following command by giving the arguments.

For arguments, first, we need to specify the test images path as a “test_data_path” argument. Second, we need to specify the recently downloaded checkpoints path as a “checkpoint_path” argument. And lastly, we need to specify the output directory path as an “output_dir” argument as shown below. This will automatically create the output directory if not present.

This will run the EAST algorithm on the test images we provided. Below an output image is shown.

In the next blog, we will explore different text detection datasets that are available. We will also learn how we can create our own text detection dataset. This will help us with training and fine-tuning our EAST model further. Till then, have a great time. Hope you enjoy reading.

If you have any doubts/suggestions please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Implementation of Efficient and Accurate Scene Text Detector (EAST)

Leave a reply

In the previous blog, we discussed the EAST algorithm, its architecture and its usage. In this blog, we will see how to implement the EAST using its GitHub Repository We will do this implementation in a Linux system.

Clone the Repository

First, you need to clone its GitHub repository on your system and change your directory to the EAST folder by using the following command.

Download Pretrained Checkpoints

Now to test this EAST model, you first need to download the pretrained checkpoints trained on ICDAR 2013 and ICDAR 2015 dataset. You can download the checkpoints from the following link:

Google Drive Link

Test the Model

After downloading pretrained checkpoints and cloning the GitHub repository, you are ready to test the model using the following command:

In the above command, you need to specify some directory paths. First, you need to specify your test image dataset path as a “test_data_path” argument. Second, you need to specify your recently downloaded checkpoints path as a “checkpoint_path” argument. And lastly, you need to specify your output directory path as an “output_dir” argument.

Sometimes you may end up with common adaptor and lanms error as shown in the following figure.

To solve these errors, you just need to use the following links or you can just google them.

  1. can not compile lanms
  2. running eval.py; undefined symbol: _Py_ZeroStruct

Running EAST using WEB

We can also run a demo by using the run_demo_server.py file provided by the GitHub repository. We just need to run the following command:

As you can see demo server is running on default port number 8769. Now you just need to open your web browser and submit the following URL:

http://localhost:8769/

Then upload the image and click on the submit button. After processing, you will see the results something like this.

Hope you enjoy reading.

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Referenced GitHub Repository: EAST: An Efficient and Accurate Scene Text Detector

Efficient and Accurate Scene Text Detector (EAST)

Leave a reply

Before the introduction of deep learning in the field of text detection, it was difficult for most text segmentation approaches to perform on challenging scenarios. Conventional approaches use manually designed features while deep learning methods learn effective features from training data. These conventional approaches are usually multi-staged which ends with slightly lesser overall performance. In this blog, we will learn a deep learning-based algorithm (EAST) that detects text with a single neural network with the elimination of multi-stage approaches.

Source

Introduction

The EAST algorithm uses a single neural network to predict a word or line-level text. It can detect text in arbitrary orientation with quadrilateral shapes. In 2017 this algorithm outperformed state of the art methods. This algorithm consists of a fully convolutional network with a non-max suppression (NMS) merging state. The fully convolutional network is used to localize text in the image and this NMS stage is basically used to merge many imprecise detected text boxes into a single bounding box for every text region (word or line text).

EAST Network Architecture

The EAST architecture was created while taking different sizes of word regions into account. The idea was to detect large word regions that require features from the later stage of the neural network while detecting small word regions that require low-level features from initial stages. To create this network, authors have used three branches combining into a single neural network.

EAST

1. Feature Extractor Stem

This branch of the network is used to extract features from different layers of the network. This stem can be a convolutional network pretrained on the ImageNet dataset. Authors of EAST architecture used PVANet and VGG16 both for the experiment. In this blog, we will see EAST architecture with the VGG16 network only. Let’s see the architecture of the VGG16 model.

VGG16

For the stem of architecture, it takes the output from the VGG16 model after pool2, pool3, pool4, and pool5 layers.

2. Feature Merging Branch

In this branch of the EAST network, it merges the feature outputs from a different layer of the VGG16 network. The input image is passed through the VGG16 model and outputs from different four layers of VGG16 are taken. Merging these feature maps will be computationally expensive. That’s why EAST uses a U-net architecture to merge feature maps gradually (see EAST architecture figure). Firstly, outputs after the pool5 layer are upsampled using a deconvolutional layer. Now the size of features after this layer would be equal to outputs from the pool4 layer and both are then merged into one layer. Then Conv 1×1 and Conv 3×3 are applied to fuse the information and produce the output of this merging stage.

Similarly outputs from other layers of the VGG16 model are concatenated and finally, a Conv 3×3 layer is applied to produce the final feature map layer before the output layer.

3. Output Layer

The output layer consists of a score map and a geometry map. The score map tells us the probability of text in that region while the geometry map defines the boundary of the text box. This geometry map can be either a rotated box or quadrangle. A rotated box consists of top-left coordinate, width, height and rotation angle for the text box. While quadrangle consists of all four coordinates of a rectangle.

Note: For more details on the Optical Character Recognition, please refer to the Mastering OCR using Deep Learning and OpenCV-Python course.

Loss Function

The loss function used in this EAST algorithm consists of both score map loss and geometry loss function.

Source

As you can see in the above formula, both losses are combined with a weight λ. This λ is for giving importance to different losses. In the EAST paper, authors have used it as 1.

Non-max Suppression Merging Stage

Predicted geometries after fully convolutional network are passed through a threshold value. After this thresholding, remaining geometries are suppressed using a locality aware NMS. A Naive NMS runs in O(n2). But to run this in O(n), authors adopted a method which uses suppression row by row. This row by row suppression also takes into account iteratively merging of the last merged one. This makes this algorithm fast in most of the cases but the worst time complexity is still O(n2).

This was all about the Efficient and Accurate Scene Text algorithm. In the next blog, we will implement this algorithm using its GitHub Repository. Hope you enjoy reading.

If you have any doubts/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.

Referenced Research Paper: EAST: An Efficient and Accurate Scene Text Detector

Optical Character Recognition Pipeline: Text Detection and Segmentation Part-II

Leave a reply

In the last blog, we have seen what is text detection and different types of algorithms to perform it, In this blog, we will learn more about text detection algorithms.

Efficient and Accurate Scene Text Detector(EAST)

It is a deep learning text detection method which has two stages one is fully convolutional network(FCN) and second is non-max suppression(NMS) merging stage. In FCN it uses U-shape network which directly produces text regions either word level or text line level. Here is the diagram of FCN used in the algorithm.

Source

U-shape FCN uses features from different layers of PVANet and then merge them to produce the outputs. The yellow boxes are different layers of PVANet and green boxes are merging layers of feature extracted from PVANet. The reason behind this merging branch is to produce outputs for both small word regions and large word regions. Low-level features will help in finding small word regions and high-level features will help in finding large word regions. This network will output geometries either in the form of RBOX(containing 5 values of which 4 are top and left coordinate, width, height and one is rotation angle) or QUAD( 4 coordinates of a rectangle) with one score map to tell about the confidence level prediction of text in it.

In second, NMS merging stage, it uses thresholding to exclude out overlapping geometries and produce the most accurate geometries for the text regions.

To implement it in our OCR pipeline, we can use it’s GitHub Repository. To make it workable use the following steps:

  1. Clone the repository into your directory: ” git clone https://github.com/argman/EAST.git”
  2. Download its pretrained model and put inside EAST directory.
  3. Before testing it you need to compile the lanms.
  4. To test this model, go to your EAST directory and then run following command from terminal:

You can also train this model with your dataset either from scratch or use pre-trained model provided earlier. To train this model you need to provide dataset path and dataset should consist of training images with corresponding text file which will have coordinates of text present in the image.

Connectionist Text Proposal Network(CTPN)

CTPN is a deep learning method that accurately predicts text lines in a natural image. It is an end to end trainable model consists of both CNN and RNN layers. In general, the length of a text line varies frequently. To solve this problem authors of this paper have considered text lines as a sequence of fine-scale text proposals, where each proposal are having a fixed width of 16 pixels with varying height. Let’s see the below image.

Source

In the above figure, each vertical rectangular box is a fine text proposal. To go through model’s architecture see below figure:

Source

The input image is being sent to VGG-16 model. Features output from conv_5 layer(the layer just before fully connected layers) of VGG-16 model is taken. A sliding window of size 3X3 is moved over VGG-16 output features and then fed sequentially to RNN network which consists of 256D bi-directional LSTM. This LSTM layer is connected to 512D fully connected layer which will next produce the outputs.

Now see the generation of output using this algorithm.

  • This algorithm uses anchor boxes to detect the text of different height. Let say we use k anchor boxes then output will consist of three main parts.
  • One is 2k vertical coordinates where each anchor box have its y coordinate (center position of box) and height of anchor box.
  • Second 2k text/non-text scores and,
  • third is k side refinement offset.
  • Here they have used 10 anchor boxes of varying height between 11 to 273 pixels. For this they have fixed the horizontal location and predicted only the vertical heights.
  • On the basis of text/non-text scores, sequential text proposal are merged and text-lines are formed. Side refinement offsets are used to refine the two end points of a text line.

To implement it in our OCR pipeline, we can use it’s GitHub Repository. To make it workable use the following steps:

  • Clone the repository into your directory: ” git clone https://github.com/eragonruan/text-detection-ctpn.git”
  • Go to “text-detection-ctpn-banjin-dev” directory
  • Run following command one by one:
  • Download pretrained checkpoint from google drive
  • Extract it and put checkpoints_mlt/ in text-detection-ctpn/
  • Now put your text file in data/demo and output will be in data/res
  • Now run the following command to check the outputs

You can also train this model using your own data, just follow the steps provide in GitHub Repository.

A Single Shot Oriented Scene Text Detector(TextBoxes++)

It is an end-to-end trainable fast scene text detector which can even detect oriented text present in the image. It does not require any post processing except non-maximum suppression. The basic idea is taken from the object detection algorithm SSD(single shot detector). SSD aims to predict general objects in an image but when it comes for text detection it fails. To improve this on text dataset TextBoxes++ have been introduced. Let’s see the model’s architecture:

Source

First 13 layers are from VGG16 model. Then 2 fully connected layers of VGG-16 are converted into convolution layers which are followed by 8 convolution layers. Finally, 6 Text-Box layers are connected to 6 different intermediate convolution layers of the model. These 6 Text-Box layers are output layer and at test time non-max separation is applied to merge the result of these 6 to predict the best ones.

Text-Box layers are the key component of TextBoxes++. These are also convolutional layer which predicts both presences of text and bounding box coordinates. It includes both oriented bounding boxes and minimum horizontal boxes. Text-Box layers are designed to tackle the problem of variable length words.

You can find it’s GitHub Repository here. In GitHub they have also implemented CRNN(convolution recurrent neural network) to recognize text detected by the TextBoxes++. To implement it, you can follow their GitHub directions. Here are some results of TextBoxes++.

Source

That’s enough for text detection, in the next blog, we will learn about text recognition. Hope you enjoy reading.

Next Blog: Optical Character Recognition Pipeline: Text Recognition

If you have any doubt/suggestion please feel free to ask and I will do my best to help or improve myself. Good-bye until next time.