A survey of human-in-the-loop for machine learning

Wu, Xingjiao; Xiao, Luwei; Sun, Yixuan; Zhang, Junhang; Ma, Tianlong; He, Liang

doi:10.1016/j.future.2022.05.014

Cited by 343 publications

(115 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Classifications were validated for specificity and sensitivity, which were highly concordant comparing human curators with the machine output [ 9, 10 ]. For the postural instability domain, the curators classified four major symptoms (sensitivity/specificity): Balance (97.0/95.4), Gait (100.0/96.5), Falling (100.0/95.2), and Freezing (100.0/91.7) [ 6 ].…”

Section: Methodsmentioning

confidence: 99%

Longitudinal Cohort Study of Verbatim-Reported Postural Instability Symptoms as Outcomes for Online Parkinson’s Disease Trials

Shoulson

Arbatti²,

Hosamath³

et al. 2022

JPD

View full text Add to dashboard Cite

Background: The Parkinson’s Disease Patient Report of Problems (PD-PROP) captures the problems and functional impact that patients report verbatim. Online research participation and advances in language analysis have enabled longitudinal collection and classification of symptoms as trial outcomes. Objective: Analyze verbatim reports longitudinally to examine postural-instability symptoms as 1) precursors of subsequent falling and 2) newly occurring symptoms that could serve as outcome measures in randomized controlled trials. Methods: Problems reported by >25,000 PD patients in their own words were collected online in the Fox Insight observational study and classified into symptoms by natural language processing, clinical curation, and machine learning. Symptoms of gait, balance, falling, and freezing and associated reports of having fallen in the last month were analyzed over three years of longitudinal observation by a Cox regression model in a cohort of 8,287 participants. New onset of gait, balance, falling, and freezing symptoms was analyzed by Kaplan-Meier survival techniques in 4,119 participants who had not previously reported these symptoms. Results: Classified verbatim symptoms of postural instability were significant precursors of subsequent falling among participants who were older, female, and had longer PD duration. New onset of symptoms steadily increased and informed sample size estimates for clinical trials to reduce the onset of these symptoms. Conclusion: The tools to analyze symptoms reported by PD patients in their own words and capacity to enroll large numbers of research participants online support the feasibility and statistical power for conducting randomized clinical trials to detect effects of therapeutic interventions.

show abstract

Section: Methodsmentioning

confidence: 99%

Longitudinal Cohort Study of Verbatim-Reported Postural Instability Symptoms as Outcomes for Online Parkinson’s Disease Trials

Shoulson

Arbatti²,

Hosamath³

et al. 2022

JPD

View full text Add to dashboard Cite

show abstract

“…Researchers have explored the human-in-the-loop paradigm (see Figure 3a) for data extraction tasks, and some efforts have been practiced in the scientific knowledge bases construction use case [48,61] to reduce the burden on researchers and improve the efficiency. Extracting and processing the unstructured "data" in the scientific literature and transferring it to structured data (from entity and attribute extraction to entity linking) is the most challenging step.…”

Section: Extract Linkmentioning

confidence: 99%

KnowledgeShovel: An AI-in-the-Loop Document Annotation System for Scientific Knowledge Base Construction

Shao¹,

Jia²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Constructing a comprehensive, accurate, and useful scientific knowledge base is crucial for human researchers synthesizing scientific knowledge and for enabling AI-driven scientific discovery. However, the current process is difficult, error-prone, and laborious due to (1) the enormous amount of scientific literature available; (2) the highly-specialized scientific domains; (3) the diverse modalities of information (text, figure, table); and, (4) the silos of scientific knowledge in different publications with inconsistent formats and structures. Informed by a formative study and iterated with participatory design workshops, we designed and developed KnowledgeShovel, an AI-in-the-Loop document annotation system for researchers to construct scientific knowledge bases. The design of KnowledgeShovel introduces a multi-step multi-modal human-AI collaboration pipeline that aligns with users' existing workflows to improve data accuracy while reducing the human burden. A follow-up user evaluation with 7 geoscience researchers shows that KnowledgeShovel can enable efficient construction of scientific knowledge bases with satisfactory accuracy.CCS Concepts: • Human-centered computing → Human computer interaction (HCI); • Applied computing → Annotation;• Computing methodologies → Information extraction.

show abstract

“…Once the model is fully built, people use it to run their scenarios. This process promotes an interplay of human interaction and Artificial Intelligence, hence following the human-in-the-loop approach that is increasingly promoted in machine learning to create more explainable models [78,79].…”

Section: Design Of the Proposed Saam Tool Overviewmentioning

confidence: 99%

Automatically Generating Scenarios from a Text Corpus: A Case Study on Electric Vehicles

2022

View full text Add to dashboard Cite

Creating ‘what-if’ scenarios to estimate possible futures is a key component of decision-making processes. However, this activity is labor intensive as it is primarily done manually by subject-matter experts who start by identifying relevant themes and their interconnections to build models, and then craft diverse and meaningful stories as scenarios to run on these models. Previous works have shown that text mining could automate the model-building aspect, for example, by using topic modeling to extract themes from a large corpus and employing variations of association rule mining to connect them in quantitative ways. In this paper, we propose to further automate the process of scenario generation by guiding pre-trained deep neural networks (i.e., BERT) through simulated conversations to extract a model from a corpus. Our case study on electric vehicles shows that our approach yields similar results to previous work while almost eliminating the need for manual involvement in model building, thus focusing human expertise on the final stage of crafting compelling scenarios. Specifically, by using the same corpus as a previous study on electric vehicles, we show that the model created here either performs similarly to the previous study when there is a consensus in the literature, or differs by highlighting important gaps on domains such as government deregulation.

show abstract

A survey of human-in-the-loop for machine learning

Cited by 343 publications

References 75 publications

Longitudinal Cohort Study of Verbatim-Reported Postural Instability Symptoms as Outcomes for Online Parkinson’s Disease Trials

Longitudinal Cohort Study of Verbatim-Reported Postural Instability Symptoms as Outcomes for Online Parkinson’s Disease Trials

KnowledgeShovel: An AI-in-the-Loop Document Annotation System for Scientific Knowledge Base Construction

Automatically Generating Scenarios from a Text Corpus: A Case Study on Electric Vehicles

Contact Info

Product

Resources

About