A Multi-Stage Deep-Learning-Based Vehicle and License Plate Recognition System with Real-Time Edge Inference

Abstract

Video streaming-based real-time vehicle identification and license plate recognition systems are challenging to design and deploy in terms of real-time processing on edge, dealing with low image resolution, high noise, and identification. This paper addresses these issues by introducing a novel multi-stage, real-time, deep learning-based vehicle identification and license plate recognition system. The system is based on a set of algorithms that efficiently integrate two object detectors, an image classifier, and a multi-object tracker to recognize car models and license plates. The information redundancy of Saudi license plates’ Arabic and English characters is leveraged to boost the license plate recognition accuracy while satisfying real-time inference performance. The system optimally achieves real-time performance on edge GPU devices and maximizes models’ accuracy by taking advantage of the temporally redundant information of the video stream’s frames. The edge device sends a notification of the detected vehicle and its license plate only once to the cloud after completing the processing. The system was experimentally evaluated on vehicles and license plates in real-world unconstrained environments at several parking entrance gates. It achieves 17.1 FPS on a Jetson Xavier AGX edge device with no delay. The comparison between the accuracy on the videos and on static images extracted from them shows that the processing of video streams using this proposed system enhances the relative accuracy of the car model and license plate recognition by 13% and 40%, respectively. This research work has won two awards in 2021 and 2022.

A Robust License Plate Detection System Using Smart Device

“…In this study, the NN technique is applied for character recognition. Multilayer perceptron, a neural network, is employed to recognize segmented characters [27]. The NN architecture for character recognition uses all segmented characters from the license plate segmentation procedure as input.…”

Efficient Multistage License Plate Detection and Recognition Using YOLOv8 and CNN for Smart Parking Systems

“…Some approaches have utilized the information redundancy of Saudi license plates' Arabic and English characters to boost the accuracy of license plate recognition while maintaining real-time inference performance. For example, Ammar et al [25] introduced a multistage, real-time, deep learningbased system for vehicle identification and license plate recognition that achieved detection rates of 81.9% and 80% and recognition rates of 67% and 95% on two videos of vehicles and license plates at several parking entrance gates. Khan et al [26] presented a deep learning-based license plate recognition system that utilizes bilingual text in license plates to restore noise-affected missing or misidentified characters in the plate with an accuracy of 99.5% for character recognition and 97% for plate detection.…”

PNANet: Probabilistic Two-Stage Detector Using Pyramid Non-Local Attention

“…Libra RCNN [ 22 ] and gradient harmonizing mechanism (GHM RCNN) [ 23 ] proposed new loss functions, optimizing the performance of detectors across different scales, difficulty levels, and object categories. Ammar et al [ 24 ] enhanced models’ accuracy by expoiting the temporally redundant information. Two-stage object detectors still achieve high accuracy nowadays, but their efficiency is low due to weak proposal generators that generate numerous but low-quality proposals [ 3 ].…”