Comprehensive Information Integration Modeling Framework for Video Titling

Zhang, Shengyu; Tan, Ziqi; Zhao, Zhou; Yu, Jianhua; Kuang, Kun; Jiang, Tan; Zhou, Jingren; Yang, Hongxia; Wu, Fei

doi:10.1145/3394486.3403325

Cited by 29 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep learning has achieved great success in many applications (such as image classification [11], video processing [35], recommender systems [36]). Unfortunately, the existence of adversarial examples [24] reveals the vulnerability of deep neural networks, which hinders the practical deployment of deep learning models.…”

Section: Introductionmentioning

confidence: 99%

Analysis and Applications of Class-wise Robustness in Adversarial Training

Tian¹,

Kuang

Jiang

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

Self Cite

View full text Add to dashboard Cite

Adversarial training is one of the most effective approaches to improve model robustness against adversarial examples. However, previous works mainly focus on the overall robustness of the model, and the in-depth analysis on the role of each class involved in adversarial training is still missing. In this paper, we propose to analyze the class-wise robustness in adversarial training. First, we provide a detailed diagnosis of adversarial training on six benchmark datasets, i.e., MNIST, CIFAR-10, CIFAR-100, SVHN, STL-10 and ImageNet. Surprisingly, we find that there are remarkable robustness discrepancies among classes, leading to unbalance/unfair class-wise robustness in the robust models. Furthermore, we keep investigating the relations between classes and find that the unbalanced class-wise robustness is pretty consistent among different attack and defense methods. Moreover, we observe that the stronger attack methods in adversarial learning achieve performance improvement mainly from a more successful attack on the vulnerable classes (i.e., classes with less robustness). Inspired by these interesting findings, we design a simple but effective attack method based on the traditional PGD attack, named Temperature-PGD attack, which proposes to enlarge the robustness disparity among classes with a temperature factor on the confidence distribution of each image. Experiments demonstrate our method can achieve a higher attack rate than the PGD attack. Furthermore, from the defense perspective, we also make some modifications in the training and inference phases to improve the robustness of the most vulnerable class, so as to mitigate the large difference in class-wise robustness. We believe our work can contribute to a more comprehensive understanding of adversarial training as well as rethinking the class-wise properties in robust models.

show abstract

Section: Introductionmentioning

confidence: 99%

Analysis and Applications of Class-wise Robustness in Adversarial Training

Tian¹,

Kuang

Jiang

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

Self Cite

View full text Add to dashboard Cite

show abstract

“…E-commerce Related Tasks. Fashion-oriented tasks [19,38,40,43,44] have been recently studied due to the rapid development of ecommerce platforms. Han et al [19] propose to learn compatibility relationships among fashion items for fashion recommendation.…”

Section: Related Workmentioning

confidence: 99%

“…Han et al [19] propose to learn compatibility relationships among fashion items for fashion recommendation. Zhang et al [44] propose to generate a short title for user-generated videos. Yang et al [40] and Zhang et al [43] propose to generate detailed descriptions for product images/videos.…”

Section: Related Workmentioning

confidence: 99%

Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training

Song

Chen²,

Jin

et al. 2021

Preprint

View full text Add to dashboard Cite

Translating e-commercial product descriptions, a.k.a product-oriented machine translation (PMT), is essential to serve e-shoppers all over the world. However, due to the domain specialty, the PMT task is more challenging than traditional machine translation problems. Firstly, there are many specialized jargons in the product description, which are ambiguous to translate without the product image. Secondly, product descriptions are related to the image in more complicated ways than standard image descriptions, involving various visual aspects such as objects, shapes, colors or even subjective styles. Moreover, existing PMT datasets are small in scale to support the research. In this paper, we first construct a large-scale bilingual product description dataset called Fashion-MMT, which contains over 114k noisy and 40k manually cleaned description translations with multiple product images. To effectively learn semantic alignments among product images and bilingual texts in translation, we design a unified product-oriented cross-modal cross-lingual model (UPOC 2 ) for pre-training and fine-tuning. Experiments on the Fashion-MMT and Multi30k datasets show that our model significantly outperforms the state-of-the-art models even pre-trained on the same dataset. It is also shown to benefit more from large-scale noisy data to improve the translation quality. We will release the dataset and codes at https://github.com/syuqings/Fashion-MMT.

show abstract

“…Great success has been achieved in many other CV tasks [68], such as super-resolution [43], image restoration [69], [70], and image generation [71]- [73]. For example, DALL-E [71] trains a 12-billion parameter autoregressive transformer for zero-shot text-to-image generation, and observe improved generalization on out-of-domain datasets.…”

Section: Computer Visionmentioning

confidence: 99%

Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI

Yao¹,

Zhang²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Influenced by the great success of deep learning via cloud computing and the rapid development of edge chips, research in artificial intelligence (AI) has shifted to both of the computing paradigms, i.e., cloud computing and edge computing. In recent years, we have witnessed significant progress in developing more advanced AI models on cloud servers that surpass traditional deep learning models owing to model innovations (e.g., Transformers, Pretrained families), explosion of training data and soaring computing capabilities. However, edge computing, especially edge and cloud collaborative computing, are still in its infancy to announce their success due to the resource-constrained IoT scenarios with very limited algorithms deployed. In this survey, we conduct a systematic review for both cloud and edge AI. Specifically, we are the first to set up the collaborative learning mechanism for cloud and edge modeling with a thorough review of the architectures that enable such mechanism. We also discuss potentials and practical experiences of some on-going advanced edge AI topics including pretraining models, graph neural networks and reinforcement learning. Finally, we discuss the promising directions and challenges in this field.

show abstract

Comprehensive Information Integration Modeling Framework for Video Titling

Cited by 29 publications

References 32 publications

Analysis and Applications of Class-wise Robustness in Adversarial Training

Analysis and Applications of Class-wise Robustness in Adversarial Training

Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training

Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI

Contact Info

Product

Resources

About