Multimodal Document Image Classification

Jain, Rajiv; Wigington, Curtis

doi:10.1109/icdar.2019.00021

Cited by 46 publications

(35 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Pondenkandath et al 23 explored four applications for document classifications including handwriting styles, layout, font, and authorship using a residual network. 65 Jain and Wigington 22 fused visual features extracted using the CNN-based deep learning network and noisy semantic information obtained using OCR to identify document categories. Khan et al 26 proposed a CNN-based approach to detect mismatching ink-color in hyperspectral document images for identifying forged documents.…”

Section: Image Classification Using Cnnmentioning

confidence: 99%

Investigating coupling preprocessing with shallow and deep convolutional neural networks in document image classification

2021

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are effective for image classification, and deeper CNNs are being used to improve classification performance. Indeed, as needs increase for searchability of vast printed document image collections, powerful CNNs have been used in place of conventional image processing. However, better performances of deep CNNs come at the expense of computational complexity. Are the additional training efforts required by deeper CNNs worth the improvement in performance? Or could a shallow CNN coupled with conventional image processing (e.g., binarization and consolidation) outperform deeper CNNbased solutions? We investigate performance gaps among shallow (LeNet-5, -7, and -9), deep (ResNet-18), and very deep (ResNet-152, MobileNetV2, and EfficientNet) CNNs for noisy printed document images, e.g., historical newspapers and document images in the RVL-CDIP repository. Our investigation considers two different classification tasks: (1) identifying poems in historical newspapers and (2) classifying 16 document types in document images. Empirical results show that a shallow CNN coupled with computationally inexpensive preprocessing can have a robust response with significantly reduced training samples; deep CNNs coupled with preprocessing can outperform very deep CNNs effectively and efficiently; and aggressive preprocessing is not helpful as it could remove potentially useful information in document images.

show abstract

Section: Image Classification Using Cnnmentioning

confidence: 99%

Investigating coupling preprocessing with shallow and deep convolutional neural networks in document image classification

2021

View full text Add to dashboard Cite

show abstract

“…Meanwhile, an average ensembling method was applied to concatenate the textual and visual stream in the proposed approach. Jain and Wigington [10] proposed another method for DLA based on the multimodal feature fusion combining a feature representation of the visual and text modalities.…”

Section: Related Workmentioning

confidence: 99%

VTLayout: Fusion of Visual and Text Features for Document Layout Analysis

Li¹,

Ma²,

Pan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Documents often contain complex physical structures, which make the Document Layout Analysis (DLA) task challenging. As a preprocessing step for content extraction, DLA has the potential to capture rich information in historical or scientific documents on a large scale. Although many deep-learning-based methods from computer vision have already achieved excellent performance in detecting Figure from documents, they are still unsatisfactory in recognizing the List, Table , Text and Title category blocks in DLA. This paper proposes a VTLayout model fusing the documents' deep visual, shallow visual, and text features to localize and identify different category blocks. The model mainly includes two stages, and the three feature extractors are built in the second stage. In the first stage, the Cascade Mask R-CNN model is applied directly to localize all category blocks of the documents. In the second stage, the deep visual, shallow visual, and text features are extracted for fusion to identify the category blocks of documents. As a result, we strengthen the classification power of different category blocks based on the existing localization technique. The experimental results show that the identification capability of the VTLayout is superior to the most advanced method of DLA based on the PubLayNet dataset, and the F1 score is as high as 0.9599.

show abstract

“…It has been empirically established that the textual features have more prominence towards document image classification contrary to image classification [9], [10]. The image features improve the classification results in a multi-modal environment but cannot alone perform the job [11], [12]. However, the past researchers in the multi-modal context exerted themselves into various image networks keeping the text feature extractor almost unexplored even though the state-of-the-art in the Natural Language Processing (NLP) technology domain has seen a paradigm shift with many transformers-based models like Bidirectional Encoder Representations from Transformers (BERT).…”

Section: Introductionmentioning

confidence: 99%

A Multi-Modal Approach to Digital Document Stream Segmentation for Title Insurance Domain

et al. 2022

View full text Add to dashboard Cite

In the twenty-first century, storing and managing digital documents has become commonplace for all corporate and public sectors around the world. Physical documents are scanned in batches and stored in a digital archive as a heterogeneous document stream, referred to as a digital package. To make Robotic Process Automation (RPA) easier, it's necessary to automatically segment the document stream into a subset of independent, coherent multi-page documents by detecting the appropriate document boundary. It's a common requirement of a TI company's Automated Document Management Systems (ADMS), where business operations are automated using RPA and the goal is to extract information from digital documents with minimal user intervention. The current study proposes, evaluates, and compares a multi-modal binary classification network incorporating text and picture aspects of digital document pages to state-of-the-art baseline methodologies. Image and textual features are extracted simultaneously from the input document image by passing them through Visual Geometry Group 16 -Convolutional Neural Network (VGG16-CNN) and pre-trained Bidirectional Encoder Representations from Transformers (Legal-BERT base ) model through transfer learning respectively. Both features are finally fused and passed through a fully connected layer of Multi Layered Perceptron (MLP) to obtain the binary classification of the pages as the First Page (FP) and Other Page (OP). Real-time document image streams from production business process archive were obtained from a reputed Title Insurance (TI) company for the study. The obtained F 1 score of 97.37% and 97.15% are significantly higher than the accuracies of the considered two baseline models and well above the expected Straight Through Pass (STP) threshold defined by the process admin.

show abstract

Multimodal Document Image Classification

Cited by 46 publications

References 21 publications

Investigating coupling preprocessing with shallow and deep convolutional neural networks in document image classification

Investigating coupling preprocessing with shallow and deep convolutional neural networks in document image classification

VTLayout: Fusion of Visual and Text Features for Document Layout Analysis

A Multi-Modal Approach to Digital Document Stream Segmentation for Title Insurance Domain

Contact Info

Product

Resources

About