Multimodal Cheapfakes Detection by Utilizing Image Captioning for Global Context

La, Tuan-Vinh; Tran, Quang-Tien; Tran, Thanh-Phuc; Tran, Anh-Duy; Dang-Nguyen, Duc-Tien; Dao, Minh-Son

doi:10.1145/3512731.3534210

Cited by 17 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to the existing method for cheapfakes detection, we have proposed a method that takes advantage of attributes from the testing dataset instead of directly alternating and defines handcraft patterns based on human effort. Moreover, we have extended experiments of the same theoretical results previously described [43]. Compared to another approach, our methods achieve competitive results, which achieve equal accuracy and higher recall and F1-score.…”

Section: Discussionmentioning

confidence: 62%

Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection

Dao

et al. 2022

Algorithms

View full text Add to dashboard Cite

The explosive growth of the social media community has increased many kinds of misinformation and is attracting tremendous attention from the research community. One of the most prevalent ways of misleading news is cheapfakes. Cheapfakes utilize non-AI techniques such as unaltered images with false context news to create false news, which makes it easy and “cheap” to create and leads to an abundant amount in the social media community. Moreover, the development of deep learning also opens and invents many domains relevant to news such as fake news detection, rumour detection, fact-checking, and verification of claimed images. Nevertheless, despite the impact on and harmfulness of cheapfakes for the social community and the real world, there is little research on detecting cheapfakes in the computer science domain. It is challenging to detect misused/false/out-of-context pairs of images and captions, even with human effort, because of the complex correlation between the attached image and the veracity of the caption content. Existing research focuses mostly on training and evaluating on given dataset, which makes the proposal limited in terms of categories, semantics and situations based on the characteristics of the dataset. In this paper, to address these issues, we aimed to leverage textual semantics understanding from the large corpus and integrated with different combinations of text-image matching and image captioning methods via ANN/Transformer boosting schema to classify a triple of (image, caption1, caption2) into OOC (out-of-context) and NOOC (no out-of-context) labels. We customized these combinations according to various exceptional cases that we observed during data analysis. We evaluate our approach using the dataset and evaluation metrics provided by the COSMOS baseline. Compared to other methods, including the baseline, our method achieves the highest Accuracy, Recall, and F1 scores.

show abstract

Section: Discussionmentioning

confidence: 62%

Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection

Dao

et al. 2022

Algorithms

View full text Add to dashboard Cite

show abstract

“…On the other hand, cheap fake defines manipulated content created through more accessible methods (Paris and Donovan, 2019), e.g. changing captions or speed of videos (La et al, 2022). The term fauxtography was first coined in journalism for images manipulated to "convey a questionable (or outright false) sense of the events they seem to depict" (Cooper, 2007;Kalb and Saivetz, 2007).…”

Section: Task Formulationmentioning

confidence: 99%

“…To obtain text representations, early approaches used combinations of word2vec models (Mikolov et al, 2013), LSTMs (Hochreiter and Schmidhuber, 1997), and TF-IDF scores for n-grams (Jin et al, 2017;Tanwar and Sharma, 2020;Hou et al, 2019). More recent efforts use pretrained language models (Fung et al, 2021;Aneja et al, 2021;La et al, 2022). To encode visual data, many approaches first detect objects in visual content using a Mask R-CNN model (He et al, 2017) before extracting visual features (Aneja et al, 2021;La et al, 2022;Shang et al, 2022).…”

Section: Stage 3: Verdict Predictionmentioning

confidence: 99%

“…More recent efforts use pretrained language models (Fung et al, 2021;Aneja et al, 2021;La et al, 2022). To encode visual data, many approaches first detect objects in visual content using a Mask R-CNN model (He et al, 2017) before extracting visual features (Aneja et al, 2021;La et al, 2022;Shang et al, 2022). Visual representations for images and videos are commonly obtained using CNN models such as ResNet (He et al, 2016;Garimella and Eckles, 2020;Abdelnabi et al, 2022), VGG (Simonyan and Zisserman, 2015;Jin et al, 2017;Sabir et al, 2018), and Inception (Szegedy et al, 2015;Guera and Delp, 2018;Roy and Ekbal, 2021).…”

Section: Stage 3: Verdict Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Automated Fact-Checking: A Survey

Akhtar,

Schlichtkrull,

Guo

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Misinformation is often conveyed in multiple modalities, e.g. a miscaptioned image. Multimodal misinformation is perceived as more credible by humans, and spreads faster than its text-only counterparts. While an increasing body of research investigates automated fact-checking (AFC), previous surveys mostly focus on text. In this survey, we conceptualise a framework for AFC including subtasks unique to multimodal misinformation. Furthermore, we discuss related terms used in different communities and map them to our framework. We focus on four modalities prevalent in real-world fact-checking: text, image, audio, and video. We survey benchmarks and models, and discuss limitations and promising directions for future research.

show abstract

“…Visual content can be manipulated, for instance, via deepfakes [42] or by combining images from different contexts in a misleading format [43]. While multimodal fake news detection is a developing field, several approaches were already presented in [44,45].…”

mentioning

confidence: 99%

Multiverse: Multilingual Evidence for Fake News Detection

2023

View full text Add to dashboard Cite

The rapid spread of deceptive information on the internet can have severe and irreparable consequences. As a result, it is important to develop technology that can detect fake news. Although significant progress has been made in this area, current methods are limited because they focus only on one language and do not incorporate multilingual information. In this work, we propose Multiverse—a new feature based on multilingual evidence that can be used for fake news detection and improve existing approaches. Our hypothesis that cross-lingual evidence can be used as a feature for fake news detection is supported by manual experiments based on a set of true (legit) and fake news. Furthermore, we compared our fake news classification system based on the proposed feature with several baselines on two multi-domain datasets of general-topic news and one fake COVID-19 news dataset, showing that (in combination with linguistic features) it yields significant improvements over the baseline models, bringing additional useful signals to the classifier.

show abstract

Multimodal Cheapfakes Detection by Utilizing Image Captioning for Global Context

Cited by 17 publications

References 12 publications

Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection

Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection

Multimodal Automated Fact-Checking: A Survey

Multiverse: Multilingual Evidence for Fake News Detection

Contact Info

Product

Resources

About