Self-Supervised Representation Learning for Document Image Classification

Abstract

Supervised learning, despite being extremely effective, relies on expensive, time-consuming, and error-prone annotations. Self-supervised learning has recently emerged as a strong alternate to supervised learning in a range of different domains as collecting a large amount of unlabeled data can be achieved by simply crawling the internet. These self-supervised methods automatically discover features relevant to represent an input example by using self-defined proxy tasks. In this paper, we question the potential of commonly employed purely supervised training (starting either from ImageNet pretrained networks or pure random initialization) in contrast to self-supervised representations that can be learned directly using self-supervised representation learning methods on large document image datasets. For this purpose, we leverage a large-scale document image collection (RVL-CDIP) to train ResNet-50 image encoder using two different self-supervision methods (SimCLR and Barlow Twins). Employing a linear classifier on top of self-supervised embeddings from ResNet-50 results in an accuracy of 86.75% as compared to 71.43% from the corresponding ImageNet pretrained embeddings. Similarly, evaluating on Tobacco-3482 dataset using self-supervised embeddings from ResNet-50 yields an accuracy of 88.52% in contrast to 74.16% from the corresponding ImageNet pretrained embeddings. We show that in the case of limited labeled data, this wide gap in performance between self-supervised and fully supervised models persists even after finetuning pretrained models. However, a significant reduction in this gap is observed with an increasing amount of data including the case where the model is trained from scratch. Our results show that representations learned using self-supervised representation learning techniques are a viable option for document image classification, specifically in the context of limited labeled data, which is a usual restriction in industrial use cases.

Adaptive Automation for Customized Products

“…While supervised learning proves highly effective in various applications, it is not without drawbacks. One significant limitation is its dependency on expensive, time-consuming, and error-prone annotations for training data [30]. This reliance on labeled data can pose a bottleneck, particularly in scenarios where acquiring large-scale labeled datasets is challenging, leading to increased costs and delays in model development.…”

VisFormers—Combining Vision and Transformers for Enhanced Complex Document Classification

“…There are several advantages of our proposed VisFormers model relative to the stateof-the-art approaches. The work conducted by [45,46] achieves an accuracy of less than 90% and requires over 900-1200 s for training using primary image classification neural networks. Nevertheless, our proposed model is designed by combining the transfer learning image classification model and Transformers and obtained an accuracy of over 96%.…”

Backpropagation-free training of deep physical neural networks

“…The training algorithm is inspired by the recently proposed forward-forward algorithm ( 31 ) and local training proposals ( 38 – 41 ) in digital neural networks, which has been extended and adapted to the supervised and unsupervised model-free physical learning of PNNs. Each nonlinear physical system performs a nonlinear transformation on input data (Fig.…”

“…Local learning has been extensively studied for training digital neural networks, from early work on Hebbian contrastive learning in Hopfield models ( 35 ) to recent biologically plausible frameworks ( 31 , 34 , 36 , 37 ), blockwise BP ( 38 , 39 ), and contrastive representation learning ( 40 , 41 ). Inspired by this concept and to address the limitations of BP-based PNN training, we proposed a simple and physics-compatible PNN architecture augmented by a physical local learning (PhyLL) algorithm.…”