EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

Abstract

Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillationbased approach that learns to reconstruct heavy egocentric video clip features by combining the semantics from a sparse set of video frames with the head motion from lightweight IMU readings. We further devise a novel selfsupervised training strategy for IMU feature learning. Our method leads to significant improvements in efficiency, requiring 200× fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPIC-Kitchens datasets, where our method outperforms state-ofthe-art efficient video understanding methods.

Salient object detection in egocentric videos

“…There is a wide range of research on egocentric videos, covering topics such as human-object interactions [23], activity recognition [24][25][26], anticipation [27], video summarization [28,29], hand detection [30], parsing social interactions [31], and inferring the camera wearer's body pose [32]. Most of these works aim to evaluate behaviors over extended temporal durations.…”

“…For video images with unknown camera parameters, we crop with default parameters. Some studies [24] focusing on egocentric videos have already shown the effectiveness of camera movement for action recognition. And in Figure 4, we show the different types of correlation between camera movement and salient movement.…”

Multimodal Distillation for Egocentric Action Recognition

“…Multimodal (egocentric) video understanding. In the context of (egocentric) video understanding, several works have shown that using additional modalities at inference time significantly improves performance [25,29,33,36,43,50,56,61]. The hypothesis is intuitive -certain actions are more easily understood from specific modalities, e.g.…”