A Hybrid Multimodal Data Fusion-Based Method for Identifying Gambling Websites

Abstract

With the development of network technology, the number of gambling websites has grown dramatically, causing a threat to social stability. There are many machine learning-based methods are proposed to identify gambling websites by analyzing the URL, the text, and the images of the websites. Nevertheless, most of the existing methods ignore one important piece of information, i.e., the text within the website images. Only the visual features of images are extracted for detection, while the semantic features of texts on the images are ignored. However, these texts have key information clearly pointing to gambling websites, which can help us identify such websites more accurately. Therefore, how to fuse image and text multimodal data is a challenge that should be met.Motivated by this, in this paper, we propose a hybrid multimodal data fusion-based method for identifying gambling websites by extracting and fusing visual and semantic features of the website screenshots. First, we fine tune the pretrained ResNet34 model to train an image classifier and to extract visual features of webpage screenshots. Second, we extract textual content from webpage screenshots through the optical character recognition (OCR) technique. We use pretrained Word2Vec word vectors as the initial embedding layer and use Bi-LSTM to train a text classifier and extract semantic features of textual content on the screenshots. Third, we use self-attention to fuse the visual and semantic features and train a multimodal classifier. The prediction results of image, text, and multimodal classifiers are fused by the late fusion method to obtain the final prediction result. To demonstrate the effectiveness of the proposed method, we conduct experiments on the webpage screenshot dataset we collected. The experimental results indicate that OCR text on the webpage screenshots has strong semantic features and the proposed hybrid multimodal data fusion based method can effectively improve the performance in identifying gambling websites, with accuracy, precision, recall, and F1-score all over 99%.

MEDAL: A Multimodality-Based Effective Data Augmentation Framework for Illegal Website Identification

“…Zhao et al [18] proposed a pornographic website detection method that learns website representation features by jointly aggregating image-based, text-based, and structure-based features, formalizing pornographic website detection as a node classification task on the graph. Wang et al [20] proposed ITSA for gambling web page detection and discussed the impact of different multimodal fusion approaches on detection effectiveness. These three approaches present different ways of utilizing multimodal features.…”

“…[25] Cyberbullying Text, Image CapsNet,CNN Zhou et al [27] Porn, Gambling, Fake, etc. Text, Image, URL BERT, ResNet, LR Chen et al [17] Porn, Gambling Text, Image, URL Doc2Vec, SVM, RF Zhao et al [18] Porn Text, Image, HTML Structure HGNN Wang et al [20] Gambling OCR Text, Image LSTM, ResNet Wang et al [19] Gambling OCR Text, Image TextRNN,CNN…”

“…The current methods for identifying these illegal websites can be divided into four main types: blacklist-based [4,5], URL-based [6][7][8][9], single modal-based [10][11][12][13][14][15][16], and multimodal fusion-based methods [17][18][19][20].…”

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…In the emotion recognition domain, 28 articles are encountered; from them, 21 were in recognition [272]- [292], 2 were in analysis [293], [294], and prediction [295], [296], and 1 was in detection [297], assessment [298], and estimating [299]. In total, 17 papers were found in the security domain, where 6 were in detection [300]- [305], 4 were in identification [306]- [309], 2 were in authentication [310], [311] and recognition [312], [313], and 1 was in determination [314], verification [315] and filtering [316]. Of the 13 papers in the biometric domain, 6 were in recognition [317]- [322], 2 were in detection [323], [324], 3 were in authentication [325]- [327] and 1 was in transforming [328] and classification [329].…”

“…The remaining 57 articles were related to combined models of model-based and agnostic-based models. Seven types of the combination were found: early & late [65], [78], [80], [86], [87], [109], [118], [123], [131], [135], [151], [165], [190], [206], [234], [235], [239], [251], [255], [269], [278], [281], [297], [302], [304], [314], [350], [358], [361], [363], [403], [412], [414], early & late & hybrid [74], [117], [232], [260], [263], [284], [331], [354], [370], early & late & kernel [150], [238], early & late & neural networks [60], [67], [183], [288], [409], early & neural networks [81], [226], [375], late & hybrid [309]<...…”