Explainable AI for Multimodal Credibility Analysis: Case Study of Online Beauty Health (Mis)-Information

Abstract

One person's data or experience is another person's information" this has become the golden rule of the 21st century which has resulted in a massive reservoir of data and immense amounts of information generation. However, there is no control over the source of this information, accessibility of this information, or the quality of it, which has given rise to the presence of "misinformation". The research community has reacted by proposing frameworks and difficulties, which are helpful for (different subtasks of) recognizing misinformation. Most of these frameworks, however, fail to consider all the aspects that can contribute to making information "credible". Furthermore, a valid explanation for each considered feature's contribution to the model's decision stands missing in most work. With this in mind, the authors have attempted to produce a system that yields highly accurate decisions, thus effectively separating credible health blogs from their non-credible counterparts while providing valid user-friendly explanations. The study proposes an Explainable AI-assisted Multimodal Credibility Assessment System that examines the credibility of the platform where the blog is hosted, the credibility of the author of the blog and the credibility of the images that contribute to the blog. This novel framework contributes to the existing body of knowledge by assessing the credibility of misleading beauty blogs using multiple crucial modalities which would lead to an insightful information consumption by the users. The proposed pipeline was successfully implemented on multiple carefully curated datasets and correctly identified 274 non credible blogs out of 321 blogs with an accuracy of 97.5%, Precision of 0.973 & F1score of 0.986. Further, the Explainable AI model, with the help of several visualizations displayed the feature contributions for each blog & it's impact and magnitude in a concise comprehensible format. The framework can be further customized and applied to various domains where presence of misinformation is of high concern such as pharmaceutical drug information, pandemic management, financial advisories, online healthcare services and cyber frauds.

Detecting and Mitigating the Dissemination of Fake News: Challenges and Future Research Opportunities

“…These framework systems can be further extended towards other domains like political agenda, education, health misinformation etc., and should take into perspective the credibility of author's claims as well to detect misinformation. [104]…”

Social media-based COVID-19 sentiment classification model using Bi-LSTM

“…These results were confirmed with a sentiment classifier accuracy of 80.33%. Additionally, Wagle, Kaur, Kamat, Patil, and Kotecha (2021) proposed a system for explaining why one particular piece of information is classified as misinformation by building an explainable AI-assisted multimodal credibility assessment system. Their method is characterized by rating the credibility of misleading forums or blogs using multiple crucial modalities which would lead to an insight information consumption by users.…”

Multimodal analysis of disinformation and misinformation

“…(I) Direct analysis of the detection model; only one paper [106] explicitly did this, which provided insight into what aspects of the data their model was reacting to (II) Examination of the dataset(s) properties mainly constrained to dataset creation papers (see next section), though estimates for biases are not always performed; the models may be configured to present data statistics, for instance presenting the levels of inter-modality interaction [99], the discordance of each modality [107], or the model's modality weightings [108]. (III) Visualizing the decision; visualizing data and the model can offer insight to researchers, but aside from [109] this was rarely attempted…”

“…The three main approaches we encountered were: Direct analysis of the detection model; only one paper [ 106 ] explicitly did this, which provided insight into what aspects of the data their model was reacting to Examination of the dataset(s) properties mainly constrained to dataset creation papers (see next section), though estimates for biases are not always performed; the models may be configured to present data statistics, for instance presenting the levels of inter-modality interaction [ 99 ], the discordance of each modality [ 107 ], or the model’s modality weightings [ 108 ]. Visualizing the decision; visualizing data and the model can offer insight to researchers, but aside from [ 109 ] this was rarely attempted Particularly for (II) and (III), domain experts can be invaluable for finding clues and patterns; this again suggests an interdisciplinary approach may provide new insights. Lastly, we note that for (II), there has been no work systematically exploring biases (and if these biases may be themselves be multimodal) and their effects on disinformation detection.…”

“…Visualizing the decision; visualizing data and the model can offer insight to researchers, but aside from [ 109 ] this was rarely attempted…”