A Challenging Dataset for Bias Detection: The Case of the Crisis in the Ukraine

Cremisini, Andres; Aguilar, Daniela; Finlayson, Mark A.

doi:10.1007/978-3-030-21741-9_18

Cited by 9 publications

(23 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ukraine Crisis Articles [6] 4,538 news articles Bias: "Pro-Russian, " "Pro-West, " "Neutral" Experts SemEval 2019 [10] 750,645 news articles Article bias, overall publisher bias Experts, Crowdworkers NewsWCL50 [9] 50 news articles Target concepts, frame properties Experts Starbucks [11] 1,235 news sentences Bias w.r.t. a reference article Crowdworkers NFNJ [12] 966 news sentences Bias w.r.t.…”

Section: Content Labels Related To Bias Labels Provided Bymentioning

confidence: 99%

“…The data set covers 2,057 sentences from 90 news articles published in 33 countries. All articles deal with the Ukraine crisis and were selected from the data set of Cremisini et al [6]. We extend this data set by 44,547 labels in total (43,197 sentence labels and 1,350 article labels).…”

Section: Content Labels Related To Bias Labels Provided Bymentioning

confidence: 99%

“…Furthermore, bias was usually not recognized in direction to any named target (e.g., Russia) [11,12], which increases ambiguity and complexity of the annotation task. Another difference is that the previous data sets were also annotated mainly on a coarse level, either on the level of news sources [10] or on the level of entire articles [6,9].…”

Section: Related Workmentioning

confidence: 99%

“…As automated approaches to news bias detection need to be evaluated and often trained, reliable data sets with ground truth annotations (i.e., labeled news articles) are of great importance. Several data sets for fake news and media bias detection exist [6,19,20]. However, due to differences in the task setup and labeling, data sets for fake news detection (e.g., [22]) are not applicable for media bias detection.…”

Section: Introductionmentioning

confidence: 99%

“…However, due to differences in the task setup and labeling, data sets for fake news detection (e.g., [22]) are not applicable for media bias detection. Furthermore, existing data sets for media bias analysis and detection have the following limitations: (1) Although a few data sets were created by experts based on an elaborate process [6], their comparably small size often limits their use. (2) Other data sets, such as the one from the SemEval-2019 Task 4 [10], are larger in size, but have rather coarse level labels, with most articles labeled according to their news source purely.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Multidimensional Dataset Based on Crowdsourcing for Analyzing and Detecting News Bias

Färber

Burkard

Jatowt

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

The automatic detection of bias in news articles can have a high impact on society because undiscovered news bias may influence the political opinions, social views, and emotional feelings of readers. While various analyses and approaches to news bias detection have been proposed, large data sets with rich bias annotations on a fine-grained level are still missing. In this paper, we firstly aggregate the aspects of news bias in related works by proposing a new annotation schema for labeling news bias. This schema covers the overall bias, as well as the bias dimensions (1) hidden assumptions, (2) subjectivity, and (3) representation tendencies. Secondly, we propose a methodology based on crowdsourcing for obtaining a large data set for news bias analysis and identification. We then use our methodology to create a data set consisting of more than 2,000 sentences annotated with 43,000 bias and bias dimension labels. Thirdly, we perform an in-depth analysis of the collected data. We show that the annotation task is difficult with respect to bias and specific bias dimensions. While crowdworkers' labels of representation tendencies correlate with experts' bias labels for articles, subjectivity and hidden assumptions do not correlate with experts' bias labels and, thus, seem to be less relevant when creating data sets with crowdworkers. The experts' article labels better match the inferred crowdworkers' article labels than the crowdworkers' sentence labels. The crowdworkers' countries of origin seem to affect their judgements. In our study, non-Western crowdworkers tend to annotate more bias either directly or in the form of bias dimensions (e.g., subjectivity) than Western crowdworkers do.

show abstract

Section: Content Labels Related To Bias Labels Provided Bymentioning

confidence: 99%

Section: Content Labels Related To Bias Labels Provided Bymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Multidimensional Dataset Based on Crowdsourcing for Analyzing and Detecting News Bias

Färber

Burkard

Jatowt

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

show abstract

Mining the online infosphere: A survey

Adak

Chakraborty

Das

et al. 2022

WIREs Data Min & Knowl

View full text Add to dashboard Cite

The evolution of Artificial Intelligence (AI)-based systems and applications have pervaded everyday life to make decisions that have a momentous impact on individuals and society. With the staggering growth of online data, often termed as the online infosphere, it has become paramount to monitor the infosphere to ensure social good as AI-based decisions are severely dependent. This survey aims to provide a comprehensive review of some of the most important research areas related to the infosphere, focusing on the technical challenges and potential solutions. The survey also outlines some of the important future directions. We begin by focussing on the collaborative systems that have emerged within the infosphere with a special thrust on Wikipedia. In the follow-up, we demonstrate how the infosphere has been instrumental in the growth of scientific citations and collaborations, thus fuelling interdisciplinary research. Finally, we illustrate the issues related to the governance of the infosphere, such as the tackling of the (a) rising hateful and abusive behavior and (b) bias and discrimination in different online platforms and news reporting.

show abstract

Media Bias Everywhere? A Vision for Dealing with the Manipulation of Public Opinion

Färber

Bartscherer

2021

Communications in Computer and Information Science

View full text Add to dashboard Cite

A Challenging Dataset for Bias Detection: The Case of the Crisis in the Ukraine

Cited by 9 publications

References 13 publications

A Multidimensional Dataset Based on Crowdsourcing for Analyzing and Detecting News Bias

A Multidimensional Dataset Based on Crowdsourcing for Analyzing and Detecting News Bias

Mining the online infosphere: A survey

Media Bias Everywhere? A Vision for Dealing with the Manipulation of Public Opinion

Contact Info

Product

Resources

About