Real-Time Brazilian License Plate Detection and Recognition Using Deep Convolutional Neural Networks

Sergio Montazzolli Silva (Home | LinkedIn | Lattes)
Claudio Rosito Jung (Home | Lattes)

Paper Information

Abstract

Automatic License Plate Recognition (ALPR) is an important task with many applications in Intelligent Transportation and Surveillance systems. As in other computer vision tasks, Deep Learning (DL) methods have been recently applied in the context of ALPR, focusing on country-specific plates, such as American or European, Chinese, Indian and Korean. However, either they are not a complete DL-ALPR pipeline, or they are commercial and utilize private datasets and lack detailed information. In this work, we proposed an end-to-end DL-ALPR system for Brazilian license plates based on state-of-the-art Convolutional Neural Network architectures. Using a publicly available dataset with Brazilian plates, the system was able to correctly detect and recognize all seven characters of a license plate in 63.18% of the test set, and 97.39% when considering at least five correct characters (partial match). Considering the segmentation and recognition of each character individually, we are able to segment 99% of the characters, and correctly recognize 93% of them.

Links and BibTex

IEEE Xplore | PDF
DOI: 10.1109/SIBGRAPI.2017.14
BibTex:

@INPROCEEDINGS{silva2017a,
author={S. M. Silva and C. R. Jung},
booktitle={2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)},
title={Real-Time Brazilian License Plate Detection and Recognition Using Deep Convolutional Neural Networks},
year={2017},
pages={55-62},
keywords={Automobiles;Cameras;Character recognition;Databases;Licenses;Training},
doi={10.1109/SIBGRAPI.2017.14},
month={Oct},}

Downloads

Frontal-View and License Plate Detection Network (FV/LPD NET)
License Plate Segmentation and Character Recogntion Network (LPS/CR NET)

Running (Linux)

For the following commands, I assume that you have the Darknet framework correctly compiled. Remember that we achieved real-time using a high-end GPU (NVIDIA Titan X).

Setting up the environment

Go to your Darknet framework folder and download the weights and configuration files (network descriptors and names) for both networks. You can use the following commands:

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/fvlpd-net.weights

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/fvlpd-net.cfg

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/fvlpd-names.txt

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/fvlpd.data

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/lpscr-net.weights

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/lpscr-net.cfg

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/lpscr-names.txt

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/lpscr.data

wget http://inf.ufrgs.br/~smsilva/real-time-brazilian-alpr/data/sample_image.jpg

Finding the car

The first step of our approach is to find the car frontal-view using the FV/LPD network. Thus, considering an example image, use the command:

./darknet detector test fvlpd.data fvlpd-net.cfg fvlpd-net.weights -thresh .4 <<< "sample_image.jpg"

As a result, you should see an image like the one below.

Detecting the License Plate

Once we have the frontal-view bounding box, you must crop it and feed the network again. Suppose we cropped the frontal-view of the second car and named it as frontal_view_car2.jpg:

./darknet detector test fvlpd.data fvlpd-net.cfg fvlpd-net.weights -thresh .2 <<< "frontal_view_car2.jpg"

The result should be...

Recognizing the License Plate

We need to crop the bounding-box of the license plate found in the previous step. Suppose our new file is lp_car2.jpg. The first thing we need to do is a small pre-processing, which consists of converting the license plate crop to grayscale. That can be done using the Linux command "convert":

convert lp_car2.jpg -colorspace Gray lp_car2_gray.jpg

Now, lets forward it on LPS/CR NET:

./darknet detector test lpscr.data lpscr-net.cfg lpscr-net.weights -thresh .6 <<< "lp_car2_gray.jpg"

And the result is...

Whoah! Characters segmented and recognized! In this particular case, the "I" was misclassified as "1".

In order to correct some misclassifications, we perform a final heuristic post-processing to exchange numbers by letters and vice versa. Actually, this (heuristic) is specific for Brazilian license plates, whose layout consists of 3 letters followed by 4 digits. Details are given in the paper.

Contact

I hope this documentation was of some help for you! However, if it was not, or you think there are some missing parts, feel free to contact me at:

Best regards!