Essay-Anchor Attentive Multi-Modal Bilinear Pooling for Textbook Question Answering

Li, Juzheng; Su, Hang; Zhu, Jun; Zhang, Bo

doi:10.1109/icme.2018.8486468

Cited by 8 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In contrast, F-GCN [12] applies graph convolutional networks [16] on textual contexts and diagrams to build unified graphs that memorize relevant question background information and predicts answers by reasoning over the graphs. EAMB [17] applies the essay-anchor attentive multi-modal bi-linear pooling method to learn the joint representations of text and diagrams. It first builds textual graphs based on textual contexts and then applies bilinear-based MFB [18] model to fuse graph and diagram representations.…”

Section: Related Workmentioning

confidence: 99%

Weakly Supervised Learning for Textbook Question Answering

Chai

Huang

et al. 2022

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Textbook Question Answering (TQA) is the task of answering diagram and non-diagram questions given large multimodal contexts consisting of abundant text and diagrams. Deep text understandings and effective learning of diagram semantics are important for this task due to its specificity. In this paper, we propose a Weakly Supervised learning method for TQA (WSTQ), which regards the incompletely accurate results of essential intermediate procedures for this task as supervision to develop Text Matching (TM) and Relation Detection (RD) tasks and then employs the tasks to motivate itself to learn strong text comprehension and excellent diagram semantics respectively. Specifically, we apply the result of text retrieval to build positive as well as negative text pairs. In order to learn deep text understandings, we first pre-train the text understanding module of WSTQ on TM and then fine-tune it on TQA. We build positive as well as negative relation pairs by checking whether there is any overlap between the items/regions detected from diagrams using object detection. The RD task forces our method to learn the relationships between regions, which are crucial to express the diagram semantics. We train WSTQ on RD and TQA simultaneously, i.e., multitask learning, to obtain effective diagram semantics and then improve the TQA performance. Extensive experiments are carried out on CK12-QA and AI2D to verify the effectiveness of WSTQ. Experimental results show that our method achieves significant accuracy improvements of 5.02% and 4.12% on test splits of the above datasets respectively than the current state-of-the-art baseline. We have released our code on https://github.com/dr-majie/WSTQ.

show abstract

Section: Related Workmentioning

confidence: 99%

Weakly Supervised Learning for Textbook Question Answering

Chai

Huang

et al. 2022

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

show abstract

“…In particular, different from using the simple label embeddings e 𝑜 and e 𝑟 in SG F , e 𝑜 is the key to connect domains of vision and language. Thus, we introduce the Multi-modal Factorized Bilinear Pooling (MFB) [45] to fuse the region features and label embeddings to augment object representation e 𝑜 , which is known to be effective in multi-modal tasks [13,16]. Node embedding : To encode the SG nodes at a unified representation u = {u 𝑜 , u 𝑎 , u 𝑟 } ∈ R 𝑢 , we introduce the Graph Convolutional Network (GCN ) [18], which can embed the graph structure into vector representations.…”

Section: Scene Graph Encodermentioning

confidence: 99%

Relational Graph Learning for Grounded Video Description Generation

Zhang

Wang

Tang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Grounded video description (GVD) encourages captioning models to attend to appropriate video regions (e.g., objects) dynamically and generate a description. Such a setting can help explain the decisions of captioning models and prevents the model from hallucinating object words in its description. However, such design mainly focuses on object word generation and thus may ignore fine-grained information and suffer from missing visual concepts. Moreover, relational words (e.g., "jump left or right") are usual spatio-temporal inference results, i.e., these words cannot be grounded on certain spatial regions. To tackle the above limitations, we design a novel relational graph learning framework for GVD, in which a language-refined scene graph representation is designed to explore fine-grained visual concepts. Furthermore, the refined graph can be regarded as relational inductive knowledge to assist captioning models in selecting the relevant information it needs to generate correct words. We validate the effectiveness of our model through automatic metrics and human evaluation, and the results indicate that our approach can generate more fine-grained and accurate description, and it solves the problem of object hallucination to some extent. CCS CONCEPTS • Computing methodologies → Scene understanding.

show abstract

“…同时, 也发展了性能良好的概率编程库, 支持贝叶斯深度学习模型的开发和部署. 例如, 我们团队研制的 "珠算" [82] Figure 9 (Color online) The architecture of image-text question answer system 为例予以说明 [85] . 如图 9 所示, 根据给定的图片, 回答以下问题, "在大陆地壳下面有多少层 (类型)?…”

Section: 深度学习方法的改进unclassified

Toward the third generation of artificial intelligence

Zhang¹,

Zhu²,

Su³

2020

Sci. Sin.-Inf.

Self Cite

View full text Add to dashboard Cite

张钹等: 迈向第三代人工智能解决范围广泛的现实问题. 由于通用方法是一种弱方法, 实际上只能解决 "玩具世界" 中的简单问题, 如机器人摆放积木, 下简单的井字棋 (tic-tac-toe) 等, 与解决复杂现实问题相差很远. 寻求通用 AI 的努力遭到了失败, 符号 AI 于 20 世纪 70 年代初跌入低谷. 幸运的是, 斯坦福大学教授费根堡姆 (E. A. Feigenbaum) 等及时改变了思路, 认为知识, 特别是特定领域的知识才是人类智能的基础, 提出知识工程 (knowledge engineering) 与专家系统 (expert systems) 等一系列强 AI 方法, 给符号 AI 带来了希望. 他们开发了专家系统 DENDRAL (有机化学结构分析系统, 1965∼1975) [6] , 随后其他学者相继开发了 MYCIN (血液传染病诊断和抗菌素处方, 1971∼1977) [7] , XCON (计算机硬件组合系统) 等. 不过早期的专家系统规模都较小, 难以实用. 直到 1997 年 5 月 IBM 的深蓝 (deep blue) 国际象棋程序打败世界冠军卡斯帕诺夫 (Kasparov), 符号 AI 才真正解决大规模复杂系统的开发问题. 费根堡姆和雷蒂 (R. Raddy) 作为设计与构造大型人工智能系统的先驱, 共同获得 1994 年 ACM 图灵奖. 符号 AI 同样可以应用于机器学习, 把 "机器学习" 看成是基于知识的 (归纳) 推理. 下面以归纳逻辑编程 (inductive logic programming, ILP) [8] 为例说明符号 AI 的学习机制. 在 ILP 中正负样本 (具体示例)、背景知识和学习结果 (假设) 都以一阶逻辑子句 (程序)

show abstract

Essay-Anchor Attentive Multi-Modal Bilinear Pooling for Textbook Question Answering

Cited by 8 publications

References 9 publications

Weakly Supervised Learning for Textbook Question Answering

Weakly Supervised Learning for Textbook Question Answering

Relational Graph Learning for Grounded Video Description Generation

Toward the third generation of artificial intelligence

Contact Info

Product

Resources

About