DocXClassifier: High Performance Explainable Deep Network for Document Image Classification

Abstract

<p>Convolutional Neural Networks (ConvNets) have been thoroughly researched for document image classification and are known for their exceptional performance in unimodal image-based document classification. Recently, however, there has been a sudden shift in the field towards multimodal approaches that simultaneously learn from the visual and textual features of the documents. While this has led to significant advances in the field, it has also led to a waning interest in improving pure ConvNets-based approaches. This is not desirable, as many of the multimodal approaches still use ConvNets as their visual backbone, and thus improving ConvNets is essential to improving these approaches. In this paper, we present DocXClassifier, a ConvNet-based approach that, using state-of-the-art model design patterns together with modern data augmentation and training strategies, not only achieves significant performance improvements in image-based document classification, but also outperforms some of the recently proposed multimodal approaches. Moreover, DocXClassifier is capable of generating transformer-like attention maps, which makes it inherently interpretable, a property not found in previous image-based classification models. Our approach achieves a new peak performance in image-based classification on two popular document datasets, namely RVL-CDIP and Tobacco3482, with a top-1 classification accuracy of 94.17% and 95.57% on the two datasets, respectively. Moreover, it sets a new record for the highest image-based classification accuracy of 90.14% on Tobacco3482 without transfer learning from RVL-CDIP. Finally, our proposed model may serve as a powerful visual backbone for future multimodal approaches, by providing much richer visual features than existing counterparts.</p>

Towards Privacy Preserved Document Image Classification - A Comprehensive Benchmark

“…In addition, the unintentional memorization [17] of training samples in these models could directly expose information about the training dataset. Surprisingly, while a plethora of research has been conducted on both document classification [5,6,35] and privacy in textual documents [13,14,18,30], we found no existing literature in the field that addresses the issue of data privacy and potential information leakage from AI-powered document image classification systems. In this work, therefore, we investigate the potential of latest privacy preservation techniques [22,23,26,28] in combination with state-of-the-art DL-based document image classification models to assess whether they can achieve sufficient utility under strong privacy constraints.…”

“…From the work of Ferrando et al (2020) [6], we investigate the EfficientNet-B4 [61], which showed the highest performance on the RVL-CDIP [34] dataset at the time. From the work of Saifullah et al (2022) [35], we investigated both ConvNext-B [2] and DocXClassifier-B [35] models, which demonstrate the current state-of-the-art performance in image-based document classification on both RVL-CDIP [34] and Tobacco3482 datasets. Finally, since Vision Transformers (ViTs) have also been explored in multiple recent studies [48][49][50][51] and show promising results, we also investigated two standard ViTs-namely, ViT-B/16 [50] and ViT-L/32 [50]-to assess their performance in comparison to the CNN architectures under private training.…”

“…To reproduce the performance of the models under non-private setting and to train the models under DP-excluded federated learning setups, we adopted the same training configurations as proposed in the original studies [5,6,35]. In particular, to train AlexNet [59], VGG-16 [60], and ResNet-50 [1], we initialized the models with Ima-geNet [62] pretrained weights and then fine-tuned them on the target document datasets (RVL-CDIP [34] or Tobacco3482) with SGD optimizer and input images resized to a fixed resolution of 224×224.…”

“…EfficientNet-B4 [61] was also trained in a similar fashion but with a multi-GPU setting and input images of resolution 384 × 384, following the approach outlined by Ferrando et al (2020) [6]. Finally, for the ConvNext-B [2] and DocXClassifier-B [35] models, we employed the training strategy proposed in [35], which involves training the models with Adam optimizer, images of resolution 384 × 384, and advanced regularization and data augmentation strategies applied during the process.…”

“…In this paper, we focus on document image classification, a fundamental component of modern document processing pipelines [5,6,33,34], typically employed at an initial stage to categorize or filter the documents prior to further processing. Numerous DL-based classification models [5][6][7][33][34][35][36], have been proposed in recent years for this task, showcasing extraordinary performance gains as compared to their traditional counterparts [37,38]. However, these models are also data-driven, relying on unaltered document images as input, and thus could easily become the target of membership inference [12] or model inversion attacks [15].…”