Hierarchical and Multimodal Classification of Images from Soil Remediation Reports

Abstract

When soil remediation specialists clean up a new site, they have a long time manually revising digital reports previously written by other experts, where they look for necessary information in accordance with similar characteristics of polluted fields. Important information lies in tables, graphs, maps, drawings and their associated captions. Therefore, experts have to be able to quickly access these content-rich elements, instead of manually scrolling through each page of entire reports. Since this information is multimodal (image and text) and follows a semantically hierarchical structure, we propose a classification algorithm that takes these two constraints into account. In contrast to existing works using either multimodal system or hierarchical classification model, we explore the combination of state-of-the-art methods from multimodal systems (image and text modalities) and hierarchical classification systems. By this combination, we tackle the constraints of our classification process: small dataset, missing modalities, noisy data, and non-English corpus. Our evaluation shows that the multimodal hierarchical system outperforms the unimodal and that the performance of multimodal system with a joint combination of hierarchical classification and flat classification on different modalities provides promising results.

Graph-Based Analysis of Hierarchical Embedding Generated by Deep Neural Network

“…The training of multi-modal and hierarchical classifier [24] for images from soil remediation reports were extended by DML to improve the embeddings quality. The obtained results using classification network was acceptable, but we faced the problem of interpreting them by the classification metrics.…”

“…The classifier [24] we have developed aims to get the embeddings from data specified by multiple modalities and hierarchical structure. As mentioned earlier, it was recently extended with DML to improve the quality of its embeddings.…”

“…We have developed a multi-modal and hierarchical classifier [24], which is able to classify images in documents from soil depollution reports. Each image is (a) described by its raw pixels, optional text caption in the document and the OCR extracted text, and (b) classified along 5 hierarchy levels defined by soil remediation experts.…”