Aligning Correlation Information for Domain Adaptation in Action Recognition

Abstract

Domain adaptation (DA) approaches address domain shift and enable networks to be applied to different scenarios. Although various image DA approaches have been proposed in recent years, there is limited research towards video DA. This is partly due to the complexity in adapting the different modalities of features in videos, which includes the correlation features extracted as long-term dependencies of pixels across spatiotemporal dimensions. The correlation features are highly associated with action classes and proven their effectiveness in accurate video feature extraction through the supervised action recognition task. Yet correlation features of the same action would differ across domains due to domain shift. Therefore we propose a novel Adversarial Correlation Adaptation Network (ACAN) to align action videos by aligning pixel correlations. ACAN aims to minimize the distribution of correlation information, termed as Pixel Correlation Discrepancy (PCD). Additionally, video DA research is also limited by the lack of cross-domain video datasets with larger domain shifts.We, therefore, introduce a novel HMDB-ARID dataset with a larger domain shift caused by a larger statistical difference between domains. This dataset is built in an effort to leverage current datasets for dark video classification.

Zero-Augmentation Pseudo Entropy for Video Unsupervised Domain Adaptation

“…In recent years, there has been a growing interest in domain adaptation, which aims to extract cross-domain shared knowledge and improve the mobility of models. Recent transfer learning methods [11][12][13], in which the main adaptation method [5,14] improves generalization of unlabeled target data by aligning distribution. It has been applied in various applications, such as image classification [32], semantic segmentation [25,33], and object detection [4,5].…”

“…We show the performance of our method in both settings. We use the official split provided by the author [14].…”

“…Many real-world applications have been made possible, such as self-driving [1,2], real-time face detection [3], human pose estimation [4,5] and fake license plate recognition [6]. The two main ingredients for successful real-world applications of the various models are (i) the emergence of large datasets on action recognition such as HMDB51 [7], UCF101 [8], Kinetics-600 [9,10]; and (ii) the establishment of transfer learning methods [11][12][13] such as unsupervised domain adaptation based on adversarial learning framework [14][15][16][17][18][19][20].…”

Multi-Source Video Domain Adaptation with Temporal Attentive Moment Alignment

“…More recently, with the wide applications of videos in various fields, there has been increasing research for Videobased Unsupervised Domain Adaptation (VUDA). The success of obtaining domain-invariant features with the above UDA methods extends to VUDA, with multiple VUDA methods proposed for tasks such as action recognition (Chen et al 2019;Pan et al 2020;Xu et al 2021b) and action segmentation (Chen et al 2020).…”

“…Though current UDA and VUDA methods (Zhang et al 2019;Xu et al 2021b) enable the transfer of knowledge across domains, they normally assume that the training Figure 1: MSVDA is more generic compared to VUDA where source data come from multiple domains, with different data distributions. MSVDA is more challenging due to the negative transfer caused by domain shifts among source domains (depicted as dashed arrow) and the need to jointly align the target domain T and the different source domains S 1 and S 2 .…”