Multi-Task Ordinal Regression for Jointly Predicting the Trustworthiness and the Leading Political Ideology of News Media

Abstract

In the context of fake news, bias, and propaganda, we study two important but relatively under-explored problems: (i) trustworthiness estimation (on a 3-point scale) and (ii) political ideology detection (left/right bias on a 7-point scale) of entire news outlets, as opposed to evaluating individual articles. In particular, we propose a multi-task ordinal regression framework that models the two problems jointly. This is motivated by the observation that hyper-partisanship is often linked to low trustworthiness, e.g., appealing to emotions rather than sticking to the facts, while center media tend to be generally more impartial and trustworthy. We further use several auxiliary tasks, modeling centrality, hyperpartisanship, as well as left-vs.-right bias on a coarse-grained scale. The evaluation results show sizable performance gains by the joint models over models that target the problems in isolation.

Timme

“…Ideology detection in general could be naturally divided into two directions, based on the targets to predict: of the politicians [7,24,28], and of the ordinary citizens [1, 2, 5, 8, 13, 15-17, 20, 23, 29]. The work conducted on ordinary citizens could also be categorized into two types according to the source of data being used: intentionally collected via strategies like survey [1,20], and directly collected such as from news articles [2] or from social networks [13,15,17]. Some studies take advantages from both sides, asking self-reported responses from a group of users selected from social networks [29], and some researchers admitted the limitations of survey experiments [23].…”

“…Some studies take advantages from both sides, asking self-reported responses from a group of users selected from social networks [29], and some researchers admitted the limitations of survey experiments [23]. Emerging from social science, probabilistic models have been widely used for such kinds of analysis since the early 1980s [2,13,28]. On the other hand, on social network datasets, it is quite intuitive trying to extract information from text data to do ideology-detection [5,8,[15][16][17], only a few paid attention to links [9,13].…”

“…For link-prediction tasks, we split all positive links into training, validation, and testing sets by 85:5:10, keeping same portion across all datasets and all models, evaluating by ROC-AUC and PR-AUC. 2…”

“…Most of their methodologies based on probabilistic models, following the long-lasting tradition started by social scientists. Some others [2,13,17,29] noticed the advantages of neural networks, but seldom do they focus on links. We will show that the social-network links' contribution to ideology detection has been under-estimated.…”

SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums

“…Dong et al (2015) proposed that a trustworthy source is one that contains very few false claims. Recent work has also focused on evaluating the factuality of reporting of entire news outlets (Baly et al, 2018(Baly et al, , 2019. 3 However, none of this work was about QA or cQA.…”