Weighted combination of per-frame recognition results for text recognition in a video stream

Abstract

The scope of uses of automated document recognition has extended and as a result, recognition techniques that do not require specialized equipment have become more relevant. Among such techniques, document recognition using mobile devices is of interest. However, it is not always possible to ensure controlled capturing conditions and, consequentially, high quality of input images. Unlike specialized scanners, mobile cameras allow using a video stream as an input, thus obtaining several images of the recognized object, captured with various characteristics. In this case, a problem of combining the information from multiple input frames arises. In this paper, we propose a weighing model for the process of combining the per-frame recognition results, two approaches to the weighted combination of the text recognition results, and two weighing criteria. The effectiveness of the proposed approaches is tested using datasets of identity documents captured with a mobile device camera in different conditions, including perspective distortion of the document image and low lighting conditions. The experimental results show that the weighting combination can improve the text recognition result quality in the video stream, and the per-character weighting method with input image focus estimation as a base criterion allows one to achieve the best results on the datasets analyzed.

Towards a unified framework for identity documents analysis and recognition

“…One of the considerations for using a video stream as an input is that it makes the input more resistant to tampering, as the video is harder to falsify in comparison with a single uploaded image, especially if the document analysis procedure is performed in real-time. From a document analysis and recognition perspective, using multiple input images of the same object presents several advantages: filtering and refinement techniques could now be employed for improving object detection and location accuracy [32,66], the so-called "super-resolution" techniques [67] could be employed for obtaining images of higher quality, and text recognition results could be improved by means of accumulating per-frame recognition results in a single most reliable one [68]. Fig.…”

“…where   is the field recognition method and   is the metric function on the set of field recognition results, both corresponding to the text field recognition problem (5) stated above. Even simple selection strategies, such as selecting a single result with the maximum value of input image quality or of a recognition result confidence level [68] can be considered a combination method, along with alignment procedures such as ROVER (Recognizer Output Voting Error Reduction) [88] and its extension for text recognition results with per-character alternatives [86].…”

“…Non-text objects such as personal signatures of photos can also undergo the process of integration T8. For example, this process may perform a selection of a single best image by analyzing the focus score [68,90], consistency of illumination or the presence of highlights [91]. Alternatively, the images extracted from individual frames may be combined into a single image of a higher quality using methods of video super-resolution [92,93].…”

A Method of Image Quality Assessment for Text Recognition on Camera-Captured and Projectively Distorted Documents

“…The results presented in [21,36] show that the presence of blur in text field images decreases the quality of recognition. It should be noted that the recognition submodule may contain a pre-processing step that refines the image using a deblurring method, for example, [37].…”

“…In [21,36], the authors incorporated an estimation of image blur into the algorithms of a combination of text recognition results in a video stream. Since an unblurred text image has high contrast in regions corresponding to strokes, they assume that the level of image blur is inversely related to the sharpness (called focus in the cited papers), which represents the directional minimum of the highest local contrasts of the image.…”