Dynamic Deep Multi-modal Fusion for Image Privacy Prediction

Tonge, Ashwini; Caragea, Cornelia

doi:10.1145/3308558.3313691

Cited by 24 publications

(16 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Buschek et al (2015) proposed a multi-modal method that assigns privacy labels to the images based on visual features and metadata like location and publication time. Tonge, Caragea, and Squicciarini (2018) utilized another kind of metadata, tag, and Tonge and Caragea (2019) further derived features of the object, scene, and tags for privacy-leaking image detection. Yang et al (2020) extracted a knowledge graph from the images and identified private images based on object detection and graph neural networks.…”

Section: Related Workmentioning

confidence: 99%

“…Yang et al focuses on objects and their correlation to identify private images based on object detection. However, they neglect other important elements like scenes (Tonge and Caragea 2019), textures, and objects beyond the capacity of pre-trained object detectors. Furthermore, the correlation among objects is fixed, but the elements vary in different images, making the fixed correlation inappropriate.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection

Yang

Cao

Shen

et al. 2022

AAAI

View full text Add to dashboard Cite

The daily practice of sharing images on social media raises a severe issue about privacy leakage. To address the issue, privacy-leaking image detection is studied recently, with the goal to automatically identify images that may leak privacy. Recent advance on this task benefits from focusing on crucial objects via pretrained object detectors and modeling their correlation. However, these methods have two limitations: 1) they neglect other important elements like scenes, textures, and objects beyond the capacity of pretrained object detectors. 2) the correlation among objects is fixed, but a fixed correlation is not appropriate for all the images. To overcome the limitations, we propose the Dynamic Region-Aware Graph Convolutional Network (DRAG) that dynamically finds out crucial regions including objects and other important elements, and model their correlation adaptively for each input image. To find out crucial regions, we cluster spatially-correlated feature channels into several region-aware feature maps. Furthermore, we dynamically model the correlation with the self-attention mechanism and explore the interaction among the regions with a graph convolutional network. The DRAG achieved an accuracy of 87% on the largest dataset for privacy-leaking image detection, which is 10 percentage points higher than the state of the art. The further case study demonstrates that it found out crucial regions containing not only objects but other important elements like textures. The code and more details are in https://github.com/guang-yanng/DRAG.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection

Yang

Cao

Shen

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…In particular, Tonge and Caragea [55] use PicAlert image dataset at one-shot to train an SVM classifier, which achieves an accuracy of 83.14% by using object tags created via ImageNet. In a more recent work [57], they propose an approach for fusing object, scene context, and image tags modalities. The model identifies the set of most competent modalities on the fly and obtains an accuracy of 86.36% .…”

Section: Performance Of the Estimation Functionmentioning

confidence: 99%

“…Although the problem of privacy setting recommendation for images have been extensively studied before, none of the approaches address all of these requirements. An important set of approaches [47,55,57,62] train various machine learning approaches to predict the privacy labels of images. The size of the data they need changes based on the trained model but many of them require a large amount of data to train accurate classifiers because of the model complexity.…”

Section: Introductionmentioning

confidence: 99%

Assisting humans in privacy management: an agent-based approach

Kurtan

Yolum

2020

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Image sharing is a service offered by many online social networks. In order to preserve privacy of images, users need to think through and specify a privacy setting for each image that they upload. This is difficult for two main reasons: first, research shows that many times users do not know their own privacy preferences, but only become aware of them over time. Second, even when users know their privacy preferences, editing these privacy settings is cumbersome and requires too much effort, interfering with the quick sharing behavior expected on an online social network. Accordingly, this paper proposes a privacy recommendation model for images using tags and an agent that implements this, namely pelte. Each user agent makes use of the privacy settings that its user have set for previous images to predict automatically the privacy setting for an image that is uploaded to be shared. When in doubt, the agent analyzes the sharing behavior of other users in the user’s network to be able to recommend to its user about what should be considered as private. Contrary to existing approaches that assume all the images are available to a centralized model, pelte is compatible to distributed environments since each agent accesses only the privacy settings of the images that the agent owner has shared or those that have been shared with the user. Our simulations on a real-life dataset shows that pelte can accurately predict privacy settings even when a user has shared a few images with others, the images have only a few tags or the user’s friends have varying privacy preferences.

show abstract

“…Interestingly, the resulting model provided more consistent predictions compared to users' judgments, indicating that users might fail to follow their own privacy-related preferences. A multimodal prediction model which mixes visual content and tags is introduced in [52]. Performance is improved by exploiting predictions from neighboring photos from the user's stream.…”

Section: Related Workmentioning

confidence: 99%

Unveiling Real-Life Effects of Online Photo Sharing

Nguyen¹,

Popescu²,

Deshayes-Chossart³

2020

Preprint

View full text Add to dashboard Cite

Social networks give free access to their services in exchange for the right to exploit their users' data. Data sharing is done in an initial context which is chosen by the users. However, data are used by social networks and third parties in different contexts which are often not transparent. We propose a new approach which unveils potential effects of data sharing in impactful real-life situations. Focus is put on visual content because of its strong influence in shaping online user profiles. The approach relies on three components: (1) a set of concepts with associated situation impact ratings obtained by crowdsourcing, (2) a corresponding set of object detectors used to analyze users' photos and (3) a ground truth dataset made of 500 visual user profiles which are manually rated for each situation. These components are combined in LERV U P , a method which learns to rate visual user profiles in each situation. LERV U P exploits a new image descriptor which aggregates concept ratings and object detections at user level. It also uses an attention mechanism to boost the detections of highly-rated concepts to prevent them from being overwhelmed by low-rated ones. Performance is evaluated per situation by measuring the correlation between the automatic ranking of profile ratings and a manual ground truth. Results indicate that LERV U P is effective since a strong correlation of the two rankings is obtained. This finding indicates that providing meaningful automatic situation-related feedback about the effects of data sharing is feasible.

show abstract

Dynamic Deep Multi-modal Fusion for Image Privacy Prediction

Cited by 24 publications

References 52 publications

DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection

DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection

Assisting humans in privacy management: an agent-based approach

Unveiling Real-Life Effects of Online Photo Sharing

Contact Info

Product

Resources

About