Weakly-Supervised Multi-Person Action Recognition in 360° Videos

Abstract

The recent development of commodity 360 • cameras have enabled a single video to capture an entire scene, which endows promising potentials in surveillance scenarios. However, research in omnidirectional video analysis has lagged behind the hardware advances. In this work, we address the important problem of action recognition in topview 360 • videos. Due to the wide filed-of-view, 360 • videos usually capture multiple people performing actions at the same time. Furthermore, the appearance of people are deformed. The proposed framework first transforms top-view omnidirectional videos into panoramic videos using a calibration free method. Then spatial-temporal features are extracted using region-based 3D CNNs for action recognition. We propose a weakly-supervised method based on multiinstance multi-label learning, which trains the model to recognize and localize multiple actions in a video using only video-level action labels as supervision. We perform experiments to quantitatively validate the efficacy of the proposed method over state-of-the-art baselines and variants of our model, and qualitatively demonstrate action localization results. To enable research in this direction, we introduce the 360Action dataset. It is the first omnidirectional video dataset for multi-person action recognition with a diverse set of scenes, actors and actions. The dataset is available at https://github.com/ryukenzen/360action. * This work is mainly completed during Junnan Li's period of internship at NEC Corporation

A Hybrid two-stream approach for Multi-Person Action Recognition in TOP-VIEW 360° Videos

“…Third line of work [2,14] [15,16,17]. Recently, a deep learning based method was adopted by authors in [3], where they proposed a calibration free method to transform top-view 360 • video frames to panorama frames and used a pre-trained 3D-CNN to extract features from them. 360 • to panoramic transformation causes people near the centre to appear distorted in the panorama frames, which affects the subsequent processing and action recognition performance.…”

“…360 • to panoramic transformation causes people near the centre to appear distorted in the panorama frames, which affects the subsequent processing and action recognition performance. In our work, we adopt the projection and panorama pro-cessing method similar to [3] while introducing a new person-centric processing stream to overcome the problem of distortion.…”

“…Earlier works in action recognition from top-view 360 • videos dealt with limited data and used handcrafted features resulting in their limited performance. However, recently, the authors in [3] used deep learning methods to tackle the problem of multi-person action recognition in top-view 360 • videos by utilizing a pretrained action recognition network trained on perspective videos, and finetuning it to the newly introduced 360Action dataset [4]. Their method was based on transforming 360 • video frames to panorama frames using a global projection method and processing the panorama frames further.…”

“…However, the omnidirectional to panorama transformation distorts the people who are present near the centre 1 . This happens 1 as defined by authors in [3] centre is the point in the 360 • video frame (a) 360 • video frame to panorama transformation proposed by [3] causes the area near the centre (yellow dot) to be highly stretched and distorted.…”

“…The amount of distortion increases radially towards the centre and people present near the centre of the 360 • video frame appear highly distorted in the corresponding panorama frame. Thus, the method in [3] performs poorly in recognizing the actions of people who are near the centre. Upon visually examining the transformed panorama frames corresponding to the videos in 360Action dataset, we found that distortion of person shape occurs in the central area within a radius of about 750 pixels from the centre, for all the scenes.…”