Are Deep Models Robust against Real Distortions? A Case Study on Document Image Classification

Abstract

Deep neural networks have been extensively researched in the field of document image classification to improve classification performance and have shown excellent results. However, there is little research in this area that addresses the question of how well these models would perform in a real-world environment, where the data the models are confronted with often exhibits various types of noise or distortion. In this work, we present two separate benchmark datasets, namely RVL-CDIP-D and Tobacco3482-D, to evaluate the robustness of existing state-of-the-art document image classifiers to different types of data distortions that are commonly encountered in the real world. The proposed benchmarks are generated by inserting 21 different types of data distortions with varying severity levels into the well-known document datasets RVL-CDIP and Tobacco3482, respectively, which are then used to quantitatively evaluate the impact of the different distortion types on the performance of latest document image classifiers. In doing so, we show that while the higher accuracy models also exhibit relatively higher robustness, they still severely underperform on some specific distortions, with their classification accuracies dropping from ~90% to as low as ~40% in some cases. We also show that some of these high accuracy models perform even worse than the baseline AlexNet model in the presence of distortions, with the relative decline in their accuracy sometimes reaching as high as 300-450% that of AlexNet. The proposed robustness benchmarks are made available to the community and may aid future research in this area.

Towards Privacy Preserved Document Image Classification - A Comprehensive Benchmark

“…[35] recently introduced DocXClassifier, a state-of-the-art transformer-inspired CNN that not only attains the highest performance in image-based classification but also possesses the property of being inherently explainable. Recent works [48,49] have also explored the use of Vision Transformers (ViTs) [50] for the document image classification task but have found it challenging to surpass CNNs using basic training approaches, even on sufficiently large datasets. However, a recent study [51] has shown that extensive pretraining enables ViTs to achieve performance levels comparable to those of CNNs but at the cost of additional training and increased complexity.…”

DocXClassifier: Towards a Robust and Interpretable Deep Neural Network for Document Image Classification

“…Subsequent research has extensively used these datasets to evaluate and improve the robustness of deep learning models using various strategies [30,34]. Taking inspiration from this, Saifullah et al (2022) [11] recently introduced two robustness benchmark datasets, namely RVL-CDIP-D and Tobacco3482-D, designed for the document domain. These datasets were generated by applying 21 different types of novel distortions to generate out-of-distribution counterparts for the RVL-CDIP [17] and Tobacco3482 document datasets, respectively.…”

“…In this section, we present a quantitative evaluation of the robustness of our proposed DocXClassifier models using the two benchmark datasets, RVL-CDIP-D and Tobacco3482-D [11], as discussed in Sec. 4.1.…”

“…Recent studies have shown that DL-based systems perform poorly when faced with minor distribution shifts in the data [9,15,16], even when trained with a number of data augmentations [10]. Such distribution shifts are commonly occurring in realworld scenarios [10], especially in the document domain [11,17], where documents are often corrupted with novel distortions at test time, such as the addition of noise, blur, ink-bleed, or stain marks [11,18]. One straightforward example is mobile-captured documents, which are commonly used by the end users but may end up with transformations or noise due to varying lighting conditions [19].…”

Analyzing the Potential of Active Learning for Document Image Classification

“…As large volumes of documents are produced on a daily basis, there is an urgent need today to automate the processing of these documents to facilitate tasks such as search, retrieval, and information extraction. However, automated processing of documents can be particularly challenging for a number of reasons, including high levels of data complexity [1], large inter-class similarity and intra-class variance [2], and corruption of scanned document data with various types of distortions [3].…”