Fi-Fo Detector: Figure and Formula Detection Using Deformable Networks

Abstract

We propose a novel hybrid approach that fuses traditional computer vision techniques with deep learning models to detect figures and formulas from document images. The proposed approach first fuses the different computer vision based image representations, i.e., color transform, connected component analysis, and distance transform, termed as Fi-Fo image representation. The Fi-Fo image representation is then fed to deep models for further refined representation-learning for detecting figures and formulas from document images. The proposed approach is evaluated on a publicly available ICDAR-2017 Page Object Detection (POD) dataset and its corrected version. It produces the state-of-the-art results for formula and figure detection in document images with an f1-score of 0.954 and 0.922, respectively. Ablation study results reveal that the Fi-Fo image representation helps in achieving superior performance in comparison to raw image representation. Results also establish that the hybrid approach helps deep models to learn more discriminating and refined features.

A Survey of Graphical Page Object Detection with Deep Neural Networks

“…Fi-Fo Detector [28] Color transform, connected component analysis, distance transform applied on images that are fed to deformable pyramid network.…”

“…Younas et al [28] exploited a similar approach by employing a deformable FPN module to detect formulas and figures in document images. Instead of providing raw input images to their deformable Faster R-CNN model, the authors have proposed an image transformation method identical to [40].…”

“…Therefore, some important features are compromised during the downsampling process in the case of detecting smaller graphical page objects such as embedded formulas [58]. • Most of the current state-of-the-art methods rely on some post-processing to obtain reliable results [28]. Therefore, more generic deep learning-based solutions are required that can detect distinctive graphical page objects in a diverse environment.…”

Cascade Network with Deformable Composite Backbone for Formula Detection in Scanned Document Images

“…The authors combined deep structure prediction with a traditional approach to detecting page objects, including formulas in document images. Younas et al [20] introduced a system called Fi-Fo that detects figures and formulas in document images. The authors empirically established that deformable convolutions [42] with Feature Pyramid Networks (FPN) [24] are a better fit as compared to other object detection algorithms.…”

“…The recent success of deep learningbased methods on computer vision within the last decade also had an impact on the task of formula detection in scanned document images. Several deep learning-based formula detection approaches [17][18][19][20] have been presented in the past two years. They are mainly equipped with object detection algorithms such as Faster R-CNN [21], YOLO [22], SSD [23], and FPNs [24].…”