KnowledgeShovel: An AI-in-the-Loop Document Annotation System for Scientific Knowledge Base Construction

Abstract

Constructing a comprehensive, accurate, and useful scientific knowledge base is crucial for human researchers synthesizing scientific knowledge and for enabling AI-driven scientific discovery. However, the current process is difficult, error-prone, and laborious due to (1) the enormous amount of scientific literature available; (2) the highly-specialized scientific domains; (3) the diverse modalities of information (text, figure, table); and, (4) the silos of scientific knowledge in different publications with inconsistent formats and structures. Informed by a formative study and iterated with participatory design workshops, we designed and developed KnowledgeShovel, an AI-in-the-Loop document annotation system for researchers to construct scientific knowledge bases. The design of KnowledgeShovel introduces a multi-step multi-modal human-AI collaboration pipeline that aligns with users' existing workflows to improve data accuracy while reducing the human burden. A follow-up user evaluation with 7 geoscience researchers shows that KnowledgeShovel can enable efficient construction of scientific knowledge bases with satisfactory accuracy.CCS Concepts: • Human-centered computing → Human computer interaction (HCI); • Applied computing → Annotation;• Computing methodologies → Information extraction.

Medicine-Engineering Interdisciplinary Research Based on Bibliometric Analysis: A Case Study on Medicine-Engineering Institutional Cooperation of Shanghai Jiao Tong University

“…To better extract and manage data, we design and impletment an AI-in-the-loop system, DeepShovel [13,14,15], where researchers collaborate with AI in extracting data efficiently from literature to build a scientific knowledge base with high data quality without the dependence of data scientists…”

“…The complete work of the system including the formative study, system description, and user study can be found at [13,14,15]. DeepShovel allows users to navigate through different tabs such as Meta, Text, Table , and Map to facilitate different aspects of data extraction.…”