Journalist-in-the-Loop: Continuous Learning as a Service for Rumour Analysis

Karmakharm, Twin; Αλέτρας, Νικόλαος; Bontcheva, Kalina

doi:10.18653/v1/d19-3020

Cited by 22 publications

(14 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human-in-the-loop can be applied to improve the performance of machine learning models by integrating human knowledge and experience for data analytics [24]. For example, human can significantly reduce algorithm bias in the training and inference in terms of human feedback for various tasks in the field of natural language processing (NLP) such as text classification [27], syntactic and semantic parsing [28], topic modeling [29], text summarization [30], and sentiment analysis [31]. The general framework is shown in Figure 3.…”

Section: B Human-in-the-loop (Hitl)mentioning

confidence: 99%

Integrating Human-in-the-loop into Swarm Learning for Decentralized Fake News Detection

Dong¹,

Qian²

2022

Preprint

View full text Add to dashboard Cite

Social media has become an effective platform to generate and spread fake news that can mislead people and even distort public opinion. Centralized methods for fake news detection, however, cannot effectively protect user privacy during the process of centralized data collection for training models. Moreover, it cannot fully involve user feedback in the loop of learning detection models for further enhancing fake news detection. To overcome these challenges, this paper proposed a novel decentralized method, Human-in-the-loop Based Swarm Learning (HBSL), to integrate user feedback into the loop of learning and inference for recognizing fake news without violating user privacy in a decentralized manner. It consists of distributed nodes that are able to independently learn and detect fake news on local data. Furthermore, detection models trained on these nodes can be enhanced through decentralized model merging. Experimental results demonstrate that the proposed method outperforms the state-of-the-art decentralized method in regard of detecting fake news on a benchmark dataset.

show abstract

Section: B Human-in-the-loop (Hitl)mentioning

confidence: 99%

Integrating Human-in-the-loop into Swarm Learning for Decentralized Fake News Detection

Dong¹,

Qian²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Text Classification (TC) is a fundamental NLP task that aims to categorize a sentence/text into its corresponding category. Karmakharm et al [76] propose a rumor classification system; the core idea of this system is to obtain additional manual feedback from the journalists to retrain a more accurate machine learning model. This framework first exploits a Rumour Classification System to classify collected social media posts and sends this information back to the journalists.…”

Section: Text Classificationmentioning

confidence: 99%

“…The experimental results in the papers we investigated show that a relatively small set of human feedback can dramatically and effectively boost the model performance. For instance, the HITL technique improves the classification accuracy for both text classification and topic modeling [76,88]. Similar situations occur in dialogue and question answering where the QA systems have higher ranking metric hits [92].…”

Section: Question Answeringmentioning

confidence: 99%

A Survey of Human-in-the-loop for Machine Learning

Wu,

Xiao,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish some tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field, along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.

show abstract

“…The scope of activities includes the creation of data sets containing the claims collected from fact-checking websites, such as MultiFC [ 15 ], Liar [ 16 ], and Truth of Varying Shades [ 17 ], and the existing solutions are based on a variety of approaches, from semi-automatic knowledge graph creation [ 18 ] to choosing check-worthy claims and comparing them against verified content (ClaimBuster) [ 19 ]. The open-domain solutions or solutions used in journalism [ 20 ] are not easily transferable to the medical domain.…”

Section: Introductionmentioning

confidence: 99%

Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training Data Sets for Machine Learning

et al. 2021

View full text Add to dashboard Cite

Background The spread of false medical information on the web is rapidly accelerating. Establishing the credibility of web-based medical information has become a pressing necessity. Machine learning offers a solution that, when properly deployed, can be an effective tool in fighting medical misinformation on the web. Objective The aim of this study is to present a comprehensive framework for designing and curating machine learning training data sets for web-based medical information credibility assessment. We show how to construct the annotation process. Our main objective is to support researchers from the medical and computer science communities. We offer guidelines on the preparation of data sets for machine learning models that can fight medical misinformation. Methods We begin by providing the annotation protocol for medical experts involved in medical sentence credibility evaluation. The protocol is based on a qualitative study of our experimental data. To address the problem of insufficient initial labels, we propose a preprocessing pipeline for the batch of sentences to be assessed. It consists of representation learning, clustering, and reranking. We call this process active annotation. Results We collected more than 10,000 annotations of statements related to selected medical subjects (psychiatry, cholesterol, autism, antibiotics, vaccines, steroids, birth methods, and food allergy testing) for less than US $7000 by employing 9 highly qualified annotators (certified medical professionals), and we release this data set to the general public. We developed an active annotation framework for more efficient annotation of noncredible medical statements. The application of qualitative analysis resulted in a better annotation protocol for our future efforts in data set creation. Conclusions The results of the qualitative analysis support our claims of the efficacy of the presented method.

show abstract

Journalist-in-the-Loop: Continuous Learning as a Service for Rumour Analysis

Cited by 22 publications

References 14 publications

Integrating Human-in-the-loop into Swarm Learning for Decentralized Fake News Detection

Integrating Human-in-the-loop into Swarm Learning for Decentralized Fake News Detection

A Survey of Human-in-the-loop for Machine Learning

Active Annotation in Evaluating the Credibility of Web-Based Medical Information: Guidelines for Creating Training Data Sets for Machine Learning

Contact Info

Product

Resources

About