MPMQA: Multimodal Question Answering on Product Manuals

Abstract

Visual contents, such as illustrations and images, play a big role in product manual understanding. Existing Product Manual Question Answering (PMQA) datasets tend to ignore visual contents and only retain textual parts. In this work, to emphasize the importance of multimodal contents, we propose a Multimodal Product Manual Question Answering (MPMQA) task. For each question, MPMQA requires the model not only to process multimodal contents but also to provide multimodal answers. To support MPMQA, a large-scale dataset PM209 is constructed with human annotations, which contains 209 product manuals from 27 well-known consumer electronic brands. Human annotations include 6 types of semantic regions for manual contents and 22,021 pairs of question and answer. Especially, each answer consists of a textual sentence and related visual regions from manuals. Taking into account the length of product manuals and the fact that a question is always related to a small number of pages, MPMQA can be naturally split into two subtasks: retrieving most related pages and then generating multimodal answers. We further propose a unified model that can perform these two subtasks all together and achieve comparable performance with multiple task-specific models. The PM209 dataset is available at https://github.com/AIM3-RUC/MPMQA.

Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation

“…Intuitively, more uniformly distributed prediction (i.e., noisy correspondence) leads to higher estimated energy uncertainty (Zhang et al 2023). Therefore we select the clean samples by applying a threshold τ and the maximum similarity constraint (Qin et al 2022), i.e.,…”

Multimodality imaging differentiation of pancreatic neuroendocrine tumors and solid pseudopapillary tumors with a nomogram model: A large single-center study

“…Text-only Paper Understanding [1,3,4,20,26,27] 21,30], tables [9,25], documents [22,28,29,43] and infographic images [23], etc.…”