A Combination of Visual-Semantic Reasoning and Text Entailment-based Boosting Algorithm for Cheapfake Detection

La, Tuan-Vinh; Dao, Minh-Son; Tran, Quang-Tien; Tran, Thanh-Phuc; Tran, Anh-Duy; Dang-Nguyen, Duc-Tien

doi:10.1145/3503161.3551595

Cited by 11 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Detecting out-of-context misinformation has been extensively researched [1,[6][7][8], and sustained efforts have been made to develop solutions for it [9][10][11][12][13][14][15][16][17]. Many datasets have been developed for this problem, including the MAIM [18], Twitter-COMMs [19], NewsCLIPpings [20] and the COSMOS dataset.…”

Section: Cheapfakes Detectionmentioning

confidence: 99%

“…Among these, the COS-MOS dataset and a self-supervised learning algorithm with the same name [1] was introduced on the Grand Challenge on detecting cheapfakes and have promoted research into this problem. Many later methods have relied on the COSMOS baseline, with several improvements proposed to enhance the results, including integrating a pretrained model for the Natural Language Inference (NLI) task [10][11][12] and using the Large Language model [13]. However, these methods rely on the testing set to finetune the models and improve their performance.…”

Section: Cheapfakes Detectionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal scene-graph matching for cheapfakes detection

Nguyen,

Dao

et al. 2024

Preprint

View full text Add to dashboard Cite

The development of technology and social media platforms has led to the proliferation of fake news, including the cheapfakes problem. Cheapfakes can be produced easily and spread quickly; a common type is out-of-context misinformation. It is worth investigating a proper algorithm that can efficiently detect such types of misinformation. In this work, we study a new approach to the detection problem of cheapfakes by using graphical neural networks to detect out-of-context samples using the dataset from the ICME 23 Grand Challenge on Detecting Cheapfakes. Specifically, our model utilizes scene graph matching and a language model pre-trained for the natural language inference task to solve task 1 of the challenge. We also propose effective methods to generate labeled data from an unlabeled training set of the challenge. Our proposed method achieved an F1-score of 85.69% and an accuracy score of 85.50% on the public testing set, surpassing the baseline by 4.69% and 3.60%, respectively.

show abstract

Section: Cheapfakes Detectionmentioning

confidence: 99%

Section: Cheapfakes Detectionmentioning

confidence: 99%

Multimodal scene-graph matching for cheapfakes detection

Nguyen,

Dao

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…[21][22][23] extract textual captions and attached images through corresponding pre-trained models then concatenate and infer through a linear layer for classifying. La et al [24] utilized an image-text matching method to measure correlations between captions and images. Dimitrina et al [25] also took advantage of Google image search to enrich information (website, categories of news, and images) and then made use of TF.IDF to predict veracity.…”

Section: Verify Claim About Imagesmentioning

confidence: 99%

Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection

Dao

et al. 2022

Algorithms

Self Cite

View full text Add to dashboard Cite

The explosive growth of the social media community has increased many kinds of misinformation and is attracting tremendous attention from the research community. One of the most prevalent ways of misleading news is cheapfakes. Cheapfakes utilize non-AI techniques such as unaltered images with false context news to create false news, which makes it easy and “cheap” to create and leads to an abundant amount in the social media community. Moreover, the development of deep learning also opens and invents many domains relevant to news such as fake news detection, rumour detection, fact-checking, and verification of claimed images. Nevertheless, despite the impact on and harmfulness of cheapfakes for the social community and the real world, there is little research on detecting cheapfakes in the computer science domain. It is challenging to detect misused/false/out-of-context pairs of images and captions, even with human effort, because of the complex correlation between the attached image and the veracity of the caption content. Existing research focuses mostly on training and evaluating on given dataset, which makes the proposal limited in terms of categories, semantics and situations based on the characteristics of the dataset. In this paper, to address these issues, we aimed to leverage textual semantics understanding from the large corpus and integrated with different combinations of text-image matching and image captioning methods via ANN/Transformer boosting schema to classify a triple of (image, caption1, caption2) into OOC (out-of-context) and NOOC (no out-of-context) labels. We customized these combinations according to various exceptional cases that we observed during data analysis. We evaluate our approach using the dataset and evaluation metrics provided by the COSMOS baseline. Compared to other methods, including the baseline, our method achieves the highest Accuracy, Recall, and F1 scores.

show abstract

Leveraging Knowledge Graphs for CheapFakes Detection: Beyond Dataset Evaluation

Dao,

Zettsu

2023

2023 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)

View full text Add to dashboard Cite

A Combination of Visual-Semantic Reasoning and Text Entailment-based Boosting Algorithm for Cheapfake Detection

Cited by 11 publications

References 5 publications

Multimodal scene-graph matching for cheapfakes detection

Multimodal scene-graph matching for cheapfakes detection

Leverage Boosting and Transformer on Text-Image Matching for Cheap Fakes Detection

Leveraging Knowledge Graphs for CheapFakes Detection: Beyond Dataset Evaluation

Contact Info

Product

Resources

About