Learning Cross-Modal Contrastive Features for Video Domain Adaptation

Audio-Adaptive Activity Recognition Across Video Domains

“…Here, we focus on video domain adaptation for activity recognition. State-of-the-art visual-only solutions learn to reduce the shift in activity appearance by adversarial training [5,6,8,9,20,27,29] and self-supervised learning techniques [9,22,27,34]. While Jamal et al [20] and Munro and Damen [27] directly penalize domain specific features with an adversarial loss at every time stamp, Chen et al [5], Choi et al [9] and Pan et al [29] attend to temporal segments that contain important cues.…”

“…Self-supervised learning objectives are also incorporated in [27] and [9] to better align the features across domains by utilizing the correspondences between RGB and optical flow or the temporal order of video clips. Song et al [34] and Kim et al [22] obtain remarkable performance by contrastive learning for self-supervised learning to align the feature distributions between video domains. Instead of relying on the vision modality only, which may present large activity appearance variance, we consider the domain-invariant information within sound to help the model adapt to the visual distribution shift.…”

“…The networks are initialized with Kinetics pre-trained model weights. Such pre-training on a large dataset is common in domain adaptation, e.g., for images (ImageNet) [7,17,40,41] and videos (Sports-1M [20], Kinetics [6,9,22,27,34]). Audio encoder.…”

“…When using a single modality, the output from the activity recognizer R(•) is directly used as the final recognition prediction. On EPIC-Kitchens [27], when using both RGB and optical flow, we average the predictions from the two modalities as the final classification result, following prior works [22,27,34].…”

“…The goal of this paper is to recognize activities such as eating, sleeping or cutting under domain shift caused by change of scenery, camera viewpoint or actor, as shown in Figure 1. Existing solutions align distribution-shifted domains inside a single visual video network by adversarial training [5,20,27,29] and self-supervised learning [9,22,34]. Although successful, projecting the visual features from different source and target domains into a shared space can make the ability of the model to distinguish between classes in the target domain suffer.…”