Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers

Abstract

The massive amounts of digitized historical documents acquired over the last decades naturally lend themselves to automatic processing and exploration. Research work seeking to automatically process facsimiles and extract information thereby are multiplying with, as a first essential step, document layout analysis. If the identification and categorization of segments of interest in document images have seen significant progress over the last years thanks to deep learning techniques, many challenges remain with, among others, the use of finer-grained segmentation typologies and the consideration of complex, heterogeneous documents such as historical newspapers. Besides, most approaches consider visual features only, ignoring textual signal. In this context, we introduce a multimodal approach for the semantic segmentation of historical newspapers that combines visual and textual features. Based on a series of experiments on diachronic Swiss and Luxembourgish newspapers, we investigate, among others, the predictive power of visual and textual features and their capacity to generalize across time and sources. Results show consistent improvement of multimodal models in comparison to a strong visual baseline, as well as better robustness to high material variance.

VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations

“…CV-based methods introduce document semantics through text embedding maps, and model layout analysis as object detection or segmentation task. MFCN [39] introduces sentence granularity semantics and inserts the text embedding maps at the decision-level (end of network), while dhSegment T 2 [3] introduces character granularity semantics and inserts text embedding maps at the input-level. Though showing great success, the above methods also bear the following limitations: limited semantics used, simple modality fusion strategy and lack of relation modeling between components.…”

“…CNN is known to be good at learning deep features. However, previous multimodal layout analysis works [39,3] only apply it to extract visual features. Text embedding maps are directly used as semantic features.…”

“…Some regions (e.g.,Figure , Table) can be easily identified by visual features, while semantic features are important for separating visually similar regions (e.g.,Abstract and Paragraph). Therefore, some recent efforts try to combine both modalities [1,20,39,3]. Here we summarize them into two categories.…”

“…CV-based methods (Fig 1 (b)) model layout analysis as object detection or segmentation task, and apply a top-down strategy. They first extract visual features by convolutional neural network and introduce semantic features through text embedding maps (at sentence-level [39] or character-level [3]), which are directly concatenated as the representation of document. Then, detection or segmentation models (e.g., Mask RCNN [13]) are used to generate layout component candidates (coordinates and semantic labels).…”

Processing the structure of documents: Logical Layout Analysis of historical newspapers in French

“…Logical layout analysis systems applied to historical documents must then account for the diachronic aspect of their layouts and adapt to the changes. [22] propose a system that goes beyond usual logical labels by labelling physical block as either Serial, Weather Forecast, Death Notice and Stock Exchange Table . To do so, their system combines visual and textual features using the word-embedding representation of each word and its coordinates on the page. Their results show that combining textual and visual features provide better results in most cases than using just one of them.…”