You watch once more: a more effective CNN architecture for video spatio-temporal action localization

Abstract

The task of spatio-temporal action localization (STAL) needs to detect the action and position of individuals in the scene. Many works focus on how to improve the accuracy, but they usually ignore inference speed and practical applications. To address the above problems, we propose a new end-to-end spatio-temporal action localization network called You Watch Once More (YWOM). In this work, there are three measures proposed to improve the accuracy of positioning and recognition while guaranteeing the inference speed. First, a new feature fusion mechanism based on frequency channel attention (FCA) is proposed, which can effectively fuse the features extracted by different backbones. In addition, a new loss function is proposed to speed up the regression and convergence of the bounding box. Specifically, the SIOU regression loss function instead of the smooth L1 loss function is applied to help the model converge stably. Moreover, a lateral connection mechanism is designed to apply more backbones to our network structure. The experimental results demonstrate that YWOM can achieve online inference speed and has good performance of spatio-temporal action localization tasks. YWOM has superiority over other related works including YOWO on the UCF101-24 dataset. The frame-mAP and the video-mAP (0.2) are improved by 4.23% and 1.63% respectively.