Temporal Attentive Alignment for Large-Scale Video Domain Adaptation

Chen, Min-Hung; Kira, Zsolt; AlRegib, Ghassan; Yoo, Jae-Kwon; Chen, Ruxin; Jian, Zheng

doi:10.1109/iccv.2019.00642

Cited by 159 publications

(205 citation statements)

References 40 publications

Supporting

Mentioning

175

Contrasting

Order By: Relevance

“…For example, Sun et al (Sun, Feng, and Saenko 2016) proposed the CORrelation Alignment (CORAL) method for unsupervised domain adaptation, which minimizes domain shift by aligning the second-order statistics of source and target distributions. In more recent studies, researchers tried to adopt GANs to seek an optimal feature space to build the mapping between two domains (Hoffman et al 2018;Sankaranarayanan et al 2018;Huang et al 2018;Chen et al 2019). Hoffman et al (Hoffman et al 2018) developed an approach, called cycle-consistent adversarial domain adaptation (Cy-CADA), to guide transfer between domains according to a discriminatively trained network.…”

Section: Unsupervised Domain Adaptationmentioning

confidence: 99%

Generative Adversarial Networks for Video-to-Video Domain Adaptation

Chen¹,

Li²,

Ma³

et al. 2020

AAAI

View full text Add to dashboard Cite

Endoscopic videos from multicentres often have different imaging conditions, e.g., color and illumination, which make the models trained on one domain usually fail to generalize well to another. Domain adaptation is one of the potential solutions to address the problem. However, few of existing works focused on the translation of video-based data. In this work, we propose a novel generative adversarial network (GAN), namely VideoGAN, to transfer the video-based data across different domains. As the frames of a video may have similar content and imaging conditions, the proposed VideoGAN has an X-shape generator to preserve the intra-video consistency during translation. Furthermore, a loss function, namely color histogram loss, is proposed to tune the color distribution of each translated frame. Two colonoscopic datasets from different centres, i.e., CVC-Clinic and ETIS-Larib, are adopted to evaluate the performance of domain adaptation of our VideoGAN. Experimental results demonstrate that the adapted colonoscopic video generated by our VideoGAN can significantly boost the segmentation accuracy, i.e., an improvement of 5%, of colorectal polyps on multicentre datasets. As our VideoGAN is a general network architecture, we also evaluate its performance with the CamVid driving video dataset on the cloudy-to-sunny translation task. Comprehensive experiments show that the domain gap could be substantially narrowed down by our VideoGAN.

show abstract

Section: Unsupervised Domain Adaptationmentioning

confidence: 99%

Generative Adversarial Networks for Video-to-Video Domain Adaptation

Chen¹,

Li²,

Ma³

et al. 2020

AAAI

View full text Add to dashboard Cite

show abstract

“…Before deep-learning, UDA for action recognition used shallow models to align source and target distributions of handcrafted features [4,11,64]. Three recent works attempted deep UDA [7,19,36]. These apply GRL adversarial training to C3D [54], TRN [63] or both [36] architectures.…”

Section: Related Workmentioning

confidence: 99%

“…Jamal et al's approach [19] outperforms shallow methods that use subspace alignment. Chen et al [7] show that attending to the temporal dynamics of videos can improve alignment. Pan et al [36] use a crossdomain attention module, to avoid uninformative frames.…”

Section: Related Workmentioning

confidence: 99%

“…Pan et al [36] use a crossdomain attention module, to avoid uninformative frames. Two of these works use RGB only [7,19] while [36] reports results on RGB and Flow, however, modalities are aligned independently and only fused during inference. The approaches [7,19,36] are evaluated on 5-7 pairs of domains from subsets of coarse-grained action recognition and gesture datasets, for example aligning UCF [41] to Olympics [34].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

Munro

Damen

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Fine-grained action recognition datasets exhibit environmental bias, where multiple video sequences are captured from a limited number of environments. Training a model in one environment and deploying in another results in a drop in performance due to an unavoidable domain shift. Unsupervised Domain Adaptation (UDA) approaches have frequently utilised adversarial training between the source and target domains. However, these approaches have not explored the multi-modal nature of video within each domain. In this work we exploit the correspondence of modalities as a self-supervised alignment approach for UDA in addition to adversarial alignment (Fig. 1).We test our approach on three kitchens from our largescale dataset, EPIC-Kitchens [8], using two modalities commonly employed for action recognition: RGB and Optical Flow. We show that multi-modal self-supervision alone improves the performance over source-only training by 2.4% on average. We then combine adversarial training with multi-modal self-supervision, showing that our approach outperforms other UDA methods by 3%.

show abstract

“…S INCE human action recognition has been widely applied in visual surveillance and some other domains [3], [5], [6], [13], [18], [25], [39], [40], [42], [46], [48], [56], thus, it has become a hot research topic. Although many action recognition algorithms [4], [18], [26], [31], [39] have been proposed, since it is a challenging task in recognizing human actions in a video, thus, these algorithms are often evaluated on laboratory environments [18], [39].…”

Section: Introductionmentioning

confidence: 99%

A Multimodal Pairwise Discrimination Network for Cross-Domain Action Recognition

et al. 2020

View full text Add to dashboard Cite

In recent years, action recognition has become a hot research topic in the computer vision and machine learning domain. Despite many well-designed action recognition approaches have been proposed, we point out that some limitations still exist including the separated fusion of different Spatiotemporal features and the reconstruction classification model, and the requirement of similar environmental conditions when capturing the training and testing data. Thus, research interest has shifted from traditional action recognition towards cross-domain action recognition. To solve these limitations, in this work, we propose a novel multimodal pairwise discrimination network (short for MPD) for cross-domain action recognition that is an end-to-end network architecture. In MPD, it can jointly fuse different Spatio-temporal features from the video, learn domain invariant features for different action domains (source and target domains), and build the classification model. To characterize the shift between these domains, subnetwork parameters in corresponding layers of MPD are required to be relevant, but not identical. Besides, the domain invariant feature discrimination needs to be improved. Extensive experimental results on two different public benchmarks including indoor environment and outdoor environment demonstrate that our MPD solution can significantly outperform state-of-the-art methods with a 4% to 20% improvement in average accuracy.

show abstract

Temporal Attentive Alignment for Large-Scale Video Domain Adaptation

Cited by 159 publications

References 40 publications

Generative Adversarial Networks for Video-to-Video Domain Adaptation

Generative Adversarial Networks for Video-to-Video Domain Adaptation

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

A Multimodal Pairwise Discrimination Network for Cross-Domain Action Recognition

Contact Info

Product

Resources

About