Privacy-Preserving Visual Content Tagging using Graph Transformer Networks

Vu, Xuan-Son; Le, Dac‐Nhuong; Edlund, Christoffer; Jiang, Lili; Nguyen, Hoang D.

doi:10.1145/3394171.3414047

Cited by 19 publications

(10 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, when DA-GAT model exploits the ResNeXt-50 [53] as the backbone, the performance of DA-GAT model is 87.1%, and it is 0.5%, 2.3% and 6.3% higher than SGTN [42] model on mAP, C-F1 (All) and O-F1 (All), respectively. Therefore, the experimental results conirm the efectiveness and superiority of DA-GAT model compared with current state-of-the-art methods.…”

Section: Methodsmentioning

confidence: 97%

“…In addition, in order to compare with the recently published literatures [3,4,55], we also exploit the image scale of 576 × 576 to train and test DA-GAT model. Meanwhile, to compare with the recently published literatures [34,42], we exploit ResNeXt-50 [53] network for image feature extraction with a semi-weakly supervised pre-trained model on ImageNet [8]. For label representations, following previous works [4,6], we leverage the 300-dim GloVe [29] model trained on the Wikipedia dataset.…”

Section: Datasets and Experimental Setingsmentioning

confidence: 99%

See 1 more Smart Citation

Double Attention Based on Graph Attention Network for Image Multi-Label Classification

Zhou

Xia

Dou

et al. 2023

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

The task of image multi-label classification is to accurately recognize multiple objects in an input image. Most of the recent works need to leverage the label co-occurrence matrix counted from training data to construct the graph structure, which are inflexible and may degrade model generalizability. In addition, these methods fail to capture the semantic correlation between the channel feature maps to further improve the model performance. To address these issues, we propose a D ouble A ttention framework based on G raph A ttention ne T work (DA-GAT) to effectively learn the correlation between labels from training data. Firstly, we devise a new channel attention mechanism to enhance the semantic correlation between channel feature maps, so as to implicitly capture the correlation between labels. Secondly, we propose a new label attention mechanism to avoid the adverse impact of manually constructed label co-occurrence matrix. It only needs to leverage the label embedding as the input of network, and then automatically constructs the label relation matrix to explicitly establish the correlation between labels. Finally, we effectively fuse the output of these two attention mechanisms to further improve the model performance. Extensive experiments are conducted on three public multi-label classification benchmarks. Our DA-GAT model achieves the mAPs of 87.1%, 96.6%, and 64.3% on MS-COCO 2014, PASCAL VOC 2007, and NUS-WIDE respectively, and obviously outperforms other existing state-of-the-art methods. In addition, visual analysis experiments demonstrate that each attention mechanism can capture the correlation between labels well, and significantly promote the model performance.

show abstract

Section: Methodsmentioning

confidence: 97%

Section: Datasets and Experimental Setingsmentioning

confidence: 99%

Double Attention Based on Graph Attention Network for Image Multi-Label Classification

Zhou

Xia

Dou

et al. 2023

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

show abstract

“…Singh et al [35] investigated human attributes (emotion, age, and gender) prediction under various de-identification privacy scenarios by body parts obfuscation. Vu et al [36] designed a framework for social media content tagging by constructing a global knowledge graph to avoid sensitive local data such as faces, passport numbers, and vehicle plates. However, these works primarily focused on static images, and none of them considered privacy issues in human motion evaluation, such as movement synchrony.…”

Section: B Privacy-preserving Machine Learningmentioning

confidence: 99%

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Li¹,

Bhat²,

Barmaki³

2022

Preprint

View full text Add to dashboard Cite

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The applications of movement synchrony are wide and broad. For example, as a measure of coordination between teammates, synchrony scores are often reported in sports. The autism community also identifies movement synchrony as a key indicator of children's social and developmental achievements. In general, raw video recordings are often used for movement synchrony estimation, with the drawback that they may reveal people's identities. Furthermore, such privacy concern also hinders data sharing, one major roadblock to a fair comparison between different approaches in autism research. To address the issue, this paper proposes an ensemble method for movement synchrony estimation, one of the first deep-learning-based methods for automatic movement synchrony assessment under privacypreserving conditions. Our method relies entirely on publicly shareable, identity-agnostic secondary data, such as skeleton data and optical flow. We validate our method on two datasets: (1) PT13 dataset collected from autism therapy interventions and (2) TASD-2 dataset collected from synchronized diving competitions. In this context, our method outperforms its counterpart approaches, both deep neural networks and alternatives.

show abstract

“…To name a few, BERT (Devlin et al 2018), RoBERTa (Liu et al 2019) are helping many language related tasks to achieve new state-of-the-art results. FastText and GloVe were tested by Chen et al (2019b), Char2Vec and BERT were experimented by Vu et al (2020). Here we investigate more into these following language models: • Char2Vec (Kim et al 2015) is a deep language model that learns at character-level inputs.…”

Section: Language Embeddingsmentioning

confidence: 99%

“…Real-world images generally embodies rich and diverse semantic information with multiple objects or actions; therefore, multi-label classification has attracted a large number of recent studies in the artificial intelligence (AI) community (Wang et al 2020a;Yeh et al 2017;Zhu et al 2017). Recognising object labels in images has many applications, ranging from social tag recommendation (Nam et al 2019;Vu et al 2020) and fashion trend analysis (Inoue et al 2017) to functional genomics (Bi and Kwok 2011). The core challenge in multi-label learning is to understand and model object dependencies to exploit attributive knowledge.…”

Section: Introductionmentioning

confidence: 99%

Modular Graph Transformer Networks for Multi-Label Image Classification

Nguyễn

2021

AAAI

Self Cite

View full text Add to dashboard Cite

With the recent advances in graph neural networks, there is a rising number of studies on graph-based multi-label classification with the consideration of object dependencies within visual data. Nevertheless, graph representations can become indistinguishable due to the complex nature of label relationships. We propose a multi-label image classification framework based on graph transformer networks to fully exploit inter-label interactions. The paper presents a modular learning scheme to enhance the classification performance by segregating the computational graph into multiple sub-graphs based on modularity. The proposed approach, named Modular Graph Transformer Networks (MGTN), is capable of employing multiple backbones for better information propagation over different sub-graphs guided by graph transformers and convolutions. We validate our framework on MS-COCO and Fashion550K datasets to demonstrate improvements for multi-label image classification. The source code is available at https://github.com/ReML-AI/MGTN.

show abstract

Privacy-Preserving Visual Content Tagging using Graph Transformer Networks

Cited by 19 publications

References 25 publications

Double Attention Based on Graph Attention Network for Image Multi-Label Classification

Double Attention Based on Graph Attention Network for Image Multi-Label Classification

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Modular Graph Transformer Networks for Multi-Label Image Classification

Contact Info

Product

Resources

About